MUSKETEER 2nd hackathon scheduled to September, 28 & 29 2021
MUSKETEER 2nd Hackathon: Attacking federated learning scenarios – Come and try to develop attacks capable of penetrating our defences in a federated learning environment.
Looking back at the 2nd Hackathon – Attacking federated learning scenarios
A hackathon is not a darkened room full of computer specialists and hackers trying to design the new Facebook, wearing hooded sweatshirts, and subsisting exclusively on caffeine and fast food. Rather, it is a modern concept where innovations are created, solutions are devised, or products are designed.
Running a hackathon is one of the most effective ways to remain relevant, grow with innovation, and enjoy exposure to the external minds in the field, specifically in the area of federated learning technologies in our case. In MUSKETEER, the fact of involving external participants ensures social diversity.
In the context of MUSKETEER, and other related IT-oriented research and development projects, it is important to organise this kind of events for one main reason: From the technical perspective, MUSKETEER creates an environment in which external users make use of the technology developed in the project scope. In this sense, you can either gather important information about how the technology is working, how the components are integrated, how the execution processes are being performed, etc. You will get very critical information in a collaborative and external environment. It is true that there are some indicators that you can get within the project members, but if you do not reach external users, you will not discover other aspects, such as which is the mature of your project development in terms of making it available for external users.
Therefore, the consortium decided to organize a second hackathon. With more than 60 registrations and 25 active participants from all over the world, the 2nd Musketeer hackathon gave us a lot of satisfaction! Over two days there was informative input from project members on the topics of federated and machine learning, tutorials and many hacking minutes.
“The event went well with the participants being engaged and learning a lot. The participants tried label flipping attacks, and then scaling the malicious weights to overcome the averaging effect.One participant contacted me on LinkedIn afterwards so was positive to have after-hackathon discussion. For constructive feedback for future it would have been beneficial in terms of organisation and delivery to have had a more accurate idea of the participant numbers and the number of teams beforehand. The IBM team was patched together after the event had started so the mentors were going through the material for the first time essentially in parallel to the participants.”
Giulio Zizzo – Research Scientist, IBM Research
We addressed mainly how good is the integration among our MUSKETEER components when external users are making use of them: pycloudmessenger + MMLL + Attacks and defences. We realised that the integration was good, and everything worked perfectly. However, we needed to consider the time that an external developer uses in order to configure their computer to be ready for our federated learning training. Other technical aspect addressed is the mature level of our MLLL and defences implemented. In the time given to the participant in order to perform the attacks, both the defences and MMLL had a very good behaviour in front of the participants attacks.
In this virtual hackathon we brought people together to work on our development and create innovative solutions under our MUSKETEER project branding. Finally, when running a hackathon, we are doing more than simply running an event. We are giving our project a very good opportunity to be popular and innovative. A hackathon provides an environment to explore what our community has to offer. We learn about industry developments while generating fresh ideas from the brightest talent.
“Artificial Intelligence Team from GMV is very glad to have participated in the 2ndHackathon: Attacking federated learning scenarios. We are aware about of the importance of security in Machine Learning models, namely in Federated learning Scenarios. This hackathon has given us the opportunity to validate our research in this amazing subfield of the Artificial Intelligence breaking the security of the proposed models.
In addition, thanks to this event we have been able to know what are the latest researches in Privacy Enhancing Technology. New security models are appearing every day and it is very important to be familiar with them in order to know their strengths and weaknesses.”
Juan Miguel Auñón García – Data Scientist, GMV [part of the winner-team Madrid]
About this Event
Date
September 28 & 29, 2021
Registration link
MUSKETEER | 2nd HACKATHON: Attacking Federated Learning Scenarios – Sep 28 | Hopin
Description
You will have the opportunity to attack our MUSKETEER platform and try to penetrate our defences and affect our federated learning training in 3 different scenarios.
SCENARIO 0
No defences. Participants will work on developing and implementing poisoning attacks on their own for a scenario with a number of honest clients and malicious users in a no defences scenario.
SCENARIO 1
Participants will work on developing and implementing poisoning attacks on their own for a scenario with honest clients and malicious users. In front there will be a defence method performed by the consortium members.
SCENARIO 2
Participants will work on developing and implementing poisoning attacks on their own. The participants will perform a black box attack against our system.
COVID-19 Update: the current crisis does not allow us to properly welcome the participants in a physical meeting. Therefore, this hackathon will be 100% online. We know the importance of meeting in person, especially for such a format of event. Therefore, we’ll do our best to organize and foster interaction among participants during the two days.
Entry skills
- Solid Python 3 programming skills, experience with training simple classifier models in Keras, basic understanding of federated learning.
- Familiarity with federated learning will be ideal.
- Participants are required to bring their own laptop.
Agenda
Hackathon Agenda 1st day – Date: Tuesday, September 28, 2021
10:00-10:10 – Welcome and introduction (TBD)
Objectives: Welcome and high level MUSKETEER introduction.
10:10-10:30 – Technical Talk: Introduction to federated learning in MUSKETEER
Objectives: Organisers provide a general introduction to Federated Learning in general (concepts, roles etc.) and the research under the MUSKETEER project in particular. General talk, intended to motivate the importance of FL, introduce general concepts / terminology, provide intuition about the working of the platform.
10:30-10:50 – Technical Talk: Introduction to MUSKETEER attacks and defences
Objectives: Organisers provide a general introduction to poisoning attacks and defences in general (concepts, roles etc.) and the research under the MUSKETEER project in particular. General talk, intended to motivate the importance of a robust environment, introduce general concepts / terminology, provide intuition about the working of the platform.
10:50-11:00 – Hackathon rules, guidelines, general instructions, Q&A
Objectives: Rules with details on number of teams, overview of the agenda, details of communication channels, a walk though of the instructions, and logistics of evaluation, the assignment of group mentors and details on breakout rooms will be provided. By the end of this session, it is expected that participants should be ready to dive into the code and organisers should be ready with their breakout room setups.
11.00-11:15 – Coffee Break
11:15-13:00 – [Breakout rooms] Hacking phase 1 (Mentors & participants)
Objectives: This session will begin in the breakout rooms where assigned mentors will work with the teams to help them run the FL setups.
- Each group executes a complete FL task (Scenario 0 – No defences).
- The mentors will act as the aggregator and 8 honest clients. The participants will act as 2 malicious clients and will run FL via pycloudmessenger using fresh credentials provided to them by their mentor.
Any remaining technical issues (e.g. missing dependencies) can be resolved here (if participants attended the prep session the week before there should not be any) - Participants will work on finishing poisoning attacks on their own for a scenario with 8 honest clients and 2 malicious users. For local development and testing, they can either connect among themselves through pycloudmessenger.
13:00-14:00 – Lunch break
14:00-15:45 – [Breakout rooms] Hacking phase 2 (Mentors & participants)
Objectives: Scenario 1: (Defence method, 8 honest users, 2 malicious users)
- To evaluate the developments the mentors will run the aggregator provided with specific defence and 8 honest clients. The participants will run 2 malicious clients.
- One organizer/mentor will be available for each group, through the platform..
- Organizers internally communicate among themselves (how are all the groups doing, do we need to lower / raise the bar, provide hints etc.) via private Slack channel and/or dedicated call.
- The developments can be evaluated any time the participants want in scenario 1.
- The required, End-of-Day-1 evaluation (still on Scenario 1) will be conducted by the mentors.
15:45-16:00 – Summary of Day 1.
- Participants can share impressions, lessons learned.
- Q&A. Qualitative feedback
- Brief participants on what to expect on Day 2 (more challenging scenarios that would penalize overfitting to specific attacks)
Hackathon Agenda 2nd day – Date: Wednesday, September 29, 2021
10:00-10:15 – Recap from Day 1 and outlook on the day
10:15-12.00 – [Breakout rooms] Hacking phase 3 & Final Evaluation. (Mentors & participants)
Objectives: Scenario 2: (Defence method, 8 honest users, 2 malicious users)
- To evaluate the developments the mentors will run the aggregator and 8 honest clients. The participants will run 2 malicious clients.
- One mentor will be available for each group, also on Slack and check-in once per hour.
- Organizers communicate among themselves to know how are all the groups doing, do we need to lower / raise the bar, provide hints etc. through private Slack channel and/or dedicated call.
- The developments can be evaluated any time the participants want in scenario 1.
- Collect final evaluations on Scenario 0, 1, 2. [Decision on how to decide the final leader boards based on the results]. Also ask the participants to walk us through their solution (explain their algorithm(s)).
12.00-12:15 – Coffee Break
12:15-12:45 – Technical talk on robustness and description of Defence Scenarios used in the Hackathon
Objectives: technical talk on robustness of federated machine learning and details about the actual defences that were used.
12:45-13:00 – Assembly, winner ceremony, virtual group photo, and closing remarks
Organizers
MUSKETEER aims to develop an industrial data platform with scalable algorithms for federated and privacy-preserving machine learning techniques, detection and mitigation of model capable of fairly monetizing datasets according to their real data value. MUSKETEER is an H2020 project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 824988.
