This research aims to understand the problem of the Swiss labor market’s dual structure, focusing on foreign workers, and to track the recent trend after the outbreak of COVID-19. This research will not only lead to the understanding of the European right-wing extremism, which is recently dispersing and gaining power, but is also directly linked to human security, which incorporates freedom from “poverty.” Foreign workers are playing a crucial role in the Swiss economy; however, the Swiss have a negative attitude towards the foreign laborers within Switzerland. 70% of the foreign workers within the Swiss labor market are from the EU/EFTA nations, and they engage in high-paying and high-skilled jobs. Despite this fact, a closer analysis reveals that although workers from the nations of Western and Northern Europe relatively engage in highly educated, highly paid, and highly skilled jobs, those from the nations of Southern and Eastern Europe carry on simple or unskilled labor. Even looking at the workers from non-European nations, non-Western workers are exposed to more vulnerable conditions in the terms of wage and unemployment rate. As the economy worsened due to the outbreak of COVID-19 in 2020, the Swiss federal government expanded the flexibility of the labor market, resulting in the worsening of the situation for the workers from the vulnerable social class vis-à-vis those from the other social classes. As expected, the victim with the greatest damage was the non-Western-European workers. That is, the intensity of threat differs in accordance with the level of vulnerability within the labor market. Asymmetrical vulnerability in the dual structure of the Swiss labor market is directly linked to the asymmetrical human security (Menschensicherheit) in the process of rapid social change triggered by COVID-19.