Debian Root on ZFS or just Home on ZFS

Helmut Neukirchen, 8. May 2024

As the Debian installer does not support to install on ZFS (due to potential legal incompatibilities between their licenses), this needs to be done manually (and as usual due to non-free firmware, Wifi is not supported out of the box by Debian, so you need to have an Ethernet cable connection):

I found a good Howto to have Debian Root on ZFS:

There are minor things to be noted:

  1. ashift=12 Some suggest that for SSDs a higher value might be better, e.g., 13. Still need to monitor wearout as ZFS has been reported to kill your SSD after a year.
  2. autotrim=on It is often said that rather a Cron jobs should do this, e.g., weekly.

Note that the above Howto contains also Rescuing using a live CD that might be helpful for rescuing.

I found two further howtos, that are somewhat different, due things in a slightly different order, but are more detailed and explicitly mention implicit steps:

I followed mainly the first howto with some inspiration from the two others. E.g. I created a standard swap partition. TODO: Enable swap and hibernate.

ZFS enables to do snapshots before and after system updates
(TODO: can this be added to any apt upgrade, incl. unattended upgrades?)

Note: After having selected KDE in tasksel, the system did not get KDE started: the textual boot was stuck at the latest task before KDE would be started (some cups stuff): I did then a apt install firmware-amd-graphics and after a reboot, it worked -- not sure whether a reboot alone would have already solved the problem or not.

For my T14, I also installed the WiFi drivers.

Home on ZFS

Somehow, the above installation was very unstable on my laptop (left mouse button not working after suspend, web browser often silently terminating), so I then used the normal Debian installer and just later moved /home to ZFS. Another argument in favour of not having root on ZFS is that in case of problems, having root on ZFS might be problem as ZFS is not natively supported by Debian (but in the above links, there is also a rescue approach mentioned). As an alternative, it has been suggested if snapshots for the root filesystem are a must, rather use BTRFS for that.

https://wiki.debian.org/ZFS for how to add ZFS.

https://theorangeone.net/posts/zfs-on-home/

https://www.shernet.com/linux/zfs-home-directory/

https://askubuntu.com/questions/1081238/migrating-home-to-zfs

Applications for enrolling in M.Sc. studies with cybersecurity specialisation: deadline 15. april

Helmut Neukirchen, 10. April 2024

Currently, we are running some advertisements on social media and Icelandic news web pages in order to raise awareness of our two M.Sc. study programmes that offer specialisations in cybersecurity: M.Sc. in Computer Science/Tölvunarfræði and M.Sc. in Software Engineering/Hugbúnaðarverkfræði.

Deadline for applications is 15. April for the M.Sc. programme. If you missed that deadline: you can still apply for a related B.Sc. programme until 5. June and if you have the right qualifications (i.e. a B.Sc. degree in Computer Science or Software Engineering) change from there into the M.Sc. programme.

US Government's Cyber Safety Review Board publishes devastating report on Microsoft's Cloud security

Helmut Neukirchen, 4. April 2024

When University of Iceland switched 2018 to Microsoft Cloud for emails and other services, I warned about this and I was also concerned about the Icelandic government's cybersecurity.

At that time, our university administration argued to me that Microsoft can better protect our cybersecurity than we can do on our own. However, threat actors managed in May and June 2023 to access Microsoft Exchange Online mailboxes: while they mainly attacked US and UK government email accounts (but also others, but who exactly is not disclosed), this could apply in theory also to our emails at University of Iceland (which enables requesting and intercepting password-reset emails in order to get access to any other non-Microsoft service that offers email-based password reset) and of course to the Icelandic government that relies as well on Microsoft's cloud-based services.

While at that time, my concerns were mainly framed by what whistleblower Edward Snowden's taught us, i.e. we get eavesdropped by our friends, 6 years later, the landscape has added more awareness on the Russian aggression and Chinese and North-Korean threat actors, including the possibility to cut off the the Internet sea cables connecting Iceland to the cloud service that are all outside of Iceland. (E.g. according to a traceroute the Microsoft cloud serves that University of Iceland uses are located in the UK -- meaning that if a state actor cuts the sea cables connecting Iceland, probably the whole IT of University of Iceland is cut off because all user authentication runs via Microsoft in the UK.)

Now, the United States Secretary of Homeland Security Cyber Safety Review Board published a devastating report on Microsoft's Cloud security. As US Government-level publications are in the public domain, I provide below some extensive direct quotes (highlighting added by me) from that report:

Throughout this review, the Board identified a series of Microsoft operational and strategic decisions that collectively point to a corporate culture that deprioritized both enterprise security investments and rigorous risk management.

To drive the rapid cultural change that is needed within Microsoft, the Board believes that Microsoft’s customers would benefit from its CEO and Board of Directors directly focusing on the company’s security culture and developing and sharing publicly a plan with specific timelines to make fundamental, security-focused reforms across the company and its full suite of products.

Microsoft leadership should consider directing internal Microsoft teams to deprioritize feature developments across the company’s cloud infrastructure and product suite until substantial security improvements have been made in order to preclude competition for resources.

The Board finds that this intrusion was preventable and should never have occurred. The Board also concludes that Microsoft’s security culture was inadequate and requires an overhaul.

The Board reaches this conclusion based on:
1. the cascade of Microsoft’s avoidable errors that allowed this intrusion to succeed;
2. Microsoft’s failure to detect the compromise of its cryptographic crown jewels on its own, relying instead on a customer to reach out to identify anomalies the customer had observed;
3. the Board’s assessment of security practices at other cloud service providers, which maintained security controls that Microsoft did not;
4. Microsoft’s failure to detect a compromise of an employee's laptop from a recently acquired company prior to allowing it to connect to Microsoft’s corporate network in 2021;
5. Microsoft’s decision not to correct, in a timely manner, its inaccurate public statements about this incident, including a corporate statement that Microsoft believed it had determined the likely root cause of the intrusion when in fact, it still has not; even though Microsoft acknowledged to the Board in November 2023 that its September 6, 2023 blog post about the root cause was inaccurate, it did not update that post until March 12, 2024, as the Board was concluding its review and only after the Board’s repeated questioning about Microsoft’s plans to issue a correction;
6. the Board's observation of a separate incident, disclosed by Microsoft in January 2024, the investigation of which was not in the purview of the Board’s review, which revealed a compromise that allowed a different nation-state actor to access highly-sensitive Microsoft corporate email accounts, source code repositories, and internal systems; and
7. how Microsoft’s ubiquitous and critical products, which underpin essential services that support national security, the foundations of our economy, and public health and safety, require the company to demonstrate the highest standards of security, accountability, and transparency.

Note that this report refers to the 2023 incident and the above only bullet point 6. refers to the most recent incident from 2024 where Microsoft had to admit that other threat actors have had access to their systems and that Microsoft is unable to prevent further accesses to their system by these actors. Or with the words of Adam Meyers, a senior vice president at the cybersecurity firm Crowdstrike:

the hackers are deep inside Microsoft, and Microsoft hasn't been able to get them out in two months

We will maybe get a further report from the Cyber Safety Review Board on this new incident.

This new incident means that these actors have access to the source code of all Microsoft products, allowing them to identify zero-day exploits, i.e. vulnerabilities that are yet unknown to Microsoft, while attackers can exploit them any time and Microsoft will not have a fix for this available in time.

Note that Microsoft owns GitHub where a huge majority of open-source software is hosted that drives today's Internet and computer systems, thus making software supply chain attacks by these actors more likely by giving them the tools to compromise the software supply chains via GitHub itself.

Compare the Microsoft attitude to blanket their security mistakes with Google creating a documentary on their security incidents (HACKING GOOGLE) and using this rather as opportunity for advertisement. (Talking about advertisement: it is a completely new experience to watch that documentary on YouTube: no ads -- well just the whole documentary is the ad.)

Official Eyvör NCC-IS kickoff

Helmut Neukirchen, 4. April 2024

While Eyvör – the National Cybersecurity Coordination Centre of Iceland (NCC-IS) has in fact started earlier, we had today the official public kick off meeting. An excerpt of the agenda is below:

We presented there the Cybersecurity research and education that is jointly done at University of Iceland and Reykjavik University. I gave the presentation on our Cybersecurity M.Sc. programmes.

ICANN and the Internet -- lunch event with ICANN representatives hosted by the Computer Science department of University of Iceland.

Helmut Neukirchen, 13. March 2024

ICANN approached the Computer Science department at the University of Iceland to facilitate a meeting with students.

Among others, ICANN allocates IP addresses globally and takes care that we have top-level domain (TLD) names in the Internet such as .is -- while this is on the one hand very technical (e.g. DNSSEC, the Domain Name System Security Extensions), it can at the same time be a political challenge (e.g. after the 2022 Russian invasion of Ukraine, the Ukraine government asked ICANN to sanction Russia by revoking the .ru TLD).

Join us for an engaging lunch event on Wednesday, the 13th of March, from 12:30 PM to 3:00 PM, where we will combine the pleasure of delicious pizza with the opportunity to learn from enlightening presentations by representatives from ICANN.

Date: Wednesday, 13th March 2024

Time: 12:30-15:00

Location: Fenjamýri, Gróska (Room "Fenjamýri" is on the first floor of Gróska more or less at the location where you would end up if you drill from the Computer Science department down two floors).

ICANN, the pivotal organization responsible for ensuring a stable and secure internet infrastructure, will be shedding light on their critical work. This presentation will not only cover the foundational aspects of how ICANN operates but also delve into the pressing policy questions currently shaping the future of internet architecture.

Agenda:
Introduction to ICANN (Chris Mondini - Vice President, Stakeholder Engagement, Europe and Managing Director for Europe)

An Overview of Current Geopolitical Challenges (Nora Mari, Government and International Governmental Organisations (IGO) Engagement Manager)

Brief Introduction to DNSSEC (Gabriella Schittek, Stakeholder Engagement Director, Nordic & Central Europe)

This is a unique chance to gain insights into the behind-the-scenes efforts that make our daily internet use possible and to understand the policy challenges that could impact the internet's framework.

Join us for an afternoon of learning, networking, and, of course, pizza. See you there!

If you are a student, at University of Iceland, we kindly request that you register for the event, as seating is limited and for pizza-estimation purposes via: https://ugla.hi.is/vidburdir/SkodaVidburd.php?sid=1448&vidburdur_id=9192.

This event is in the context of Eyvör - the Cybersecurity National Coordination Centre Iceland (NCC-IS).

Computer Science department at UT messan 2024 IT fair

Helmut Neukirchen, 13. March 2024

The Computer Science department was heavily present at UT messan 2024, the biggest IT fair in Iceland. We had booths, showcasing computer games written by our students, research on using big touch screens, and cyber security demos. Also we were moderating a session and our rector was giving a talk on research that heavily involved our AI activities. Watch out for photos.

Cybersecurity specialisation now available in our Software Engineering Master's programme / Kjörsvið Netöryggi í mastersnám í hugbúnaðarverkfræði

Helmut Neukirchen, 1. March 2024

Starting from autumn 2024, we are offering the Cybersecurity specialisation now also in our Software Engineering Master's programme (so far, it was only available in Computer Science).

For details, see the course catalogue entry for the Software Engineering Master's programme / mastersnám í hugbúnaðarverkfræði (note that the set of mandatory courses is not 100% correct and still subject to change). For applying, have a look at the Icelandic web page mastersnám í hugbúnaðarverkfræði (netöryggi) or the English web page Software Engineering MSc programme (Cybersecurity). Application deadline is 15. April for students from Iceland.

If you rather want to do the Cybersecurity specialisation in our Computer Science MSc programme (it has less focus on Software Engineering and more on Computer Networks):

For details, see the course catalogue entry for the Computer Science Master's programme / mastersnám í hugbúnaðarverkfræði. For applying, have a look at the Icelandic web page mastersnám í tölvunarfræði (netöryggi) or the English web page Computer Science MSc programme (Cybersecurity). Application deadline is 15. April for students from Iceland.

I have also web page on the Collaboration for the joint cybersecurity M.Sc. programme of Reykjavik University (RU) and University of Iceland (UoI).

Cybersecurity tabletop exercise Arctic Cranes

Helmut Neukirchen, 22. February 2024

On 29.11.2023, we had a Cybersecurity tabletop exercise entitled Arctic Cranes that was arranged by the Fulbright visiting scholar Larry Leibrock with an introduction by Sigurður Emil Pálsson from the NATO Cooperative Cyber Defence Centre of Excellence (CCDCOE). In this exercise, a response to a cyber-attack on the harbour infrastructure, in particular the cranes to load and unload vessels, was trained. In the worst case, this would mean that Iceland would run out of food or medicines (a limited supply might be possible to provide via air freight, though -- at least as long as there is no cyber-attack on air traffic at the same time).
In addition to the above parties who facilitated this event, this was in the context of Eyvör/NCC-IS the National Cybersecurity Coordination Centre of Iceland (co-funded by the European Union) and the Ministry of Higher Education, Science and Innovation-funded joint cybersecurity Master's programme.

EVSE-DIN access via Modbus and mbpoll

Helmut Neukirchen, 16. December 2023

An Electric Vehicle Supply Equipment (EVSE) is the controller in a wallbox that checks how many Ampere the charging cable is able to transfer (as indicated by the PP resistor inside the cable) and communicates this via a PWM signal to the electric vehicle (EV) and when the EV signals that it wants to charge, then the EVSE finally closes the contactor inside the wallbox so that the charging current can flow to the onboard charger (OBC) inside the EV (the OBC then slowly ramps up the charging current and ramps it down at the end of charging, so contactor closing/connecting and opening/disconnecting does not involve an electric arc, so that contactor welding is not an issue).

After the Phoenix Contact EVSE in my Wallbe Eco 3.6 kW wallbox died, I replaced it with an EVSE-DIN (essentially the EVSE-WB in a DIN rail enclosure) from EV Racing. Note that there are types with a DC residual current monitoring and without. If you take the variant without, then you need to have a type B RCD.

I bought the version with Modbus from the seller OpenWB: that seller had Modbus already enabled (factory setting is disabled) and had the maximum current set to 13 A (more foolproof than the default 32 A), but had unfortunately also disabled PP detection and had it instead set to 32 Ampere (instead of measuring the PP resistor value).

I also bought an RS485 to USB converter so that I can access the EVSE-DIN's Modbus via USB.

The EVSE-DIN is very simple, but there exists a third party add-on that uses an ESP32 to add Wifi support, web server and other functionality: EVSE-WiFi. (And if you want to play around a lot with an EVSE: use an EV simulator to try out your EVSE, including checking error-handling: as any contactor, the contactors inside the EV have limited amount of open/close cycles, so using your car to test your EVSE may exhaust the cycles of the contactor inside your car).

Modbus is the protocol, RS485 is the physical layer (often, this is called RTU mode, but in fact, RTU mode means that each byte contains 8 bit of information instead of using ASCII character encoding where two ASCII characters would be needed for transmitting 8 bits) -- there exist also a TCP layer as lower layer for Modbus.

By default, 16 bit values are transmitted as values (when bit-level access is supported, then bits are called coils).

For EVSE-DIN with RS485, the baudrate is 9600 with 8N1, i.e. data bits are 8, parity is none, and stop bits is 1. The Modbus slave address of the EVSE-DIN is 1, but that can be changed via Modbus register 2001.

The registers that are accessible via Modbus can be found in the EVSE-DIN documentation.

On my Linux system, the RS485 to USB converter can be accessed via /dev/ttyUSB0 and that needs be accessed as super user.

I use mbpoll to read and write data:

To read the 10 registers from 1000 to 1009 via RS485, with addresses starting from 0 and a 1-time printout (using mbpoll's default slave address, i.e. 1):
sudo mbpoll -m rtu -b 9600 -d 8 -P none -s 1 -0 -1 /dev/ttyUSB0 -r 1000 -c 10

The result is
mbpoll 1.0-0 - FieldTalk(tm) Modbus(R) Master Simulator
Copyright © 2015-2019 Pascal JEAN, https://github.com/epsilonrt/mbpoll
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type 'mbpoll -w' for details.

Protocol configuration: Modbus RTU
Slave configuration...: address = [1]
start reference = 1000, count = 10
Communication.........: /dev/ttyUSB0, 9600-8N1
t/o 1.00 s, poll rate 1000 ms
Data type.............: 16-bit register, output (holding) register table

-- Polling slave 1...
[1000]: 13
[1001]: 0
[1002]: 1
[1003]: 32
[1004]: 0
[1005]: 18
[1006]: 1
[1007]: 0
[1008]: 0
[1009]: 0

To read the 17 registers from 2000 to 2016:
sudo mbpoll -m rtu -b 9600 -d 8 -P none -s 1 -0 -1 /dev/ttyUSB0 -r 2000 -c 17

The result is
[2000]: 13
[2001]: 1
[2002]: 5
[2003]: 1
[2004]: 0
[2005]: 521
[2006]: 0
[2007]: 32
[2008]: 65535 (-1)
[2009]: 3
[2010]: 6
[2011]: 10
[2012]: 16
[2013]: 25
[2014]: 32
[2015]: 48
[2016]: 63

When writing values, be aware that you need always to write at least two consecutive values (i.e. do a read to find out what the second value is and simply rewrite it). If you try to write only one single value to EVSE-DIN, you will get the following error:
Write output (holding) register failed: Illegal function

The reason is that Modbus function code 06 for writing a single value is not supported by EVSE-DIN.
From the above EVSE-DIN PDF:

NOTE#3: Only functions 03 (Read Holding Registers) and 16 (Preset Multiple Registers)
are implemented. For more details please check: http://www.simplymodbus.ca/FAQ.htm

Therefore, to write values, take care to write at least two values to make mbpoll use the Modbus function 16 (Preset Multiple Registers).

Changes that I did:

Set default (maximum) charging current to 17 Ampere in order to reflect the cable and circuit breaker rating in my garage (which is 16 A, but my EV is careful and uses always a little bit less, i.e. 16 A when the EVSE allows to use 17 A):
Change register 2000 from 13 to 17.

As single values cannot be written to EVSE-DIN, I had to write multiple values and the above read shows that 2001 is 1, so I wrote starting from 2000 the two values 17 1:
sudo mbpoll -m rtu -b 9600 -d 8 -P none -s 1 -0 /dev/ttyUSB0 -r 2000 17 1,

Make LED not quickly flashing when idle, but permanently on when idle (and slowly flashing when charging, i.e. contactor is on. Any remaining fast flashing is an indicator of an error):

Change bit 2 in register 2005 to gain constant LED ON while no EV is connected. Register 2005 had a value of 521, i.e. 0b1000001001 = (bit 1 set, i.e. button can be used to change charging current; bit 3 set, i.e. support to charge with ventilation for lead battery-based EVs; bit 9 set, i.e. pilot auto recover delay) and after setting bit 2, the new values is 525. Writing that value should work via

Enable PP detection (i.e. it takes the maximum charge current from the connected charging cable):

Change register 2007 from 32 to 0.

As single values cannot be written to EVSE-DIN, I wrote multiple values and the above read shows that 2006 is 0, so I wrote starting from 2005 the three values 525 0 0:

sudo mbpoll -m rtu -b 9600 -d 8 -P none -s 1 -0 /dev/ttyUSB0 -r 2005 525 0 0,
which gives

bpoll 1.0-0 - FieldTalk(tm) Modbus(R) Master Simulator
Copyright © 2015-2019 Pascal JEAN, https://github.com/epsilonrt/mbpoll
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type 'mbpoll -w' for details.

Protocol configuration: Modbus RTU
Slave configuration...: address = [1]
start reference = 2005, count = 3
Communication.........: /dev/ttyUSB0, 9600-8N1
t/o 1.00 s, poll rate 1000 ms
Data type.............: 16-bit register, output (holding) register table

Written 3 references.

A new read confirms that the changes have been applied:

sudo mbpoll -m rtu -b 9600 -d 8 -P none -s 1 -0 -1 /dev/ttyUSB0 -r 2000 -c 17
mbpoll 1.0-0 - FieldTalk(tm) Modbus(R) Master Simulator
Copyright © 2015-2019 Pascal JEAN, https://github.com/epsilonrt/mbpoll
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type 'mbpoll -w' for details.

Protocol configuration: Modbus RTU
Slave configuration...: address = [1]
start reference = 2000, count = 17
Communication.........: /dev/ttyUSB0, 9600-8N1
t/o 1.00 s, poll rate 1000 ms
Data type.............: 16-bit register, output (holding) register table

-- Polling slave 1...
[2000]: 17
[2001]: 1
[2002]: 5
[2003]: 1
[2004]: 0
[2005]: 525
[2006]: 0
[2007]: 0
[2008]: 65535 (-1)
[2009]: 3
[2010]: 6
[2011]: 10
[2012]: 16
[2013]: 25
[2014]: 32
[2015]: 48
[2016]: 63

The LED is now on when idle (instead of nervously blinking). Also, after having enabled PP detection, register 1003 has then the value 5 with no cable attached and 20 with an 20 Ampere cable attached.

In future, I might try enabling a second LED (via pin AN) as a dedicated error LED (need to find out what the normal LED does then display). Currently, the single LED is on when the wallbox is idle, i.e. no EV connected, flashes two times quickly followed by a short pause when the EV is connected (PWM generated by EVSE), but does not charge (e.g. EV finished charging or EV is initiating charging), and flashes slowly while the EV is charging, i.e. the contactor is closed.

I might also re-use the keyswitch of the Wallbe Eco case to interrupt charging: interrupting the CP signal should be the way to go...

Another future work is to upgrade the wallbox to 20 Amps and 3 phases. I already added a 3L+N contactor and the charging port cabling supports 20 Amps as well as the Phoenix terminal blocks do. Just all internal wiring needs to be upgraded to 3 phase 20 Amps.

Eyvör NCC-IS and ICEDEF cybersecurity talks join the Icelandic HPC Community Workshop

Helmut Neukirchen, 13. December 2023

The cybersecurity projects Eyvör NCC-IS and ICEDEF partnered with the Icelandic HPC Community Workshop event series.

At the 13th Icelandic HPC Community Workshop we have a couple of cybersecurity talks:

  • Geir Olav Dyrkolbotn (NTNU): Strengthening the Defence of Norway through knowledge
  • Skeggi Thormar (Upwind): eBPF and Cyber Security
  • Tom Welsh (University of Iceland): Adaptive Inspection of Industry 4.0 Supply Chains for Fraud Detection

December 13, 2023 5:00 PM, Gróska, Bjargargötu 1, 102 Reykjavik – Entrance A, 1st Floor, Room Fenjamýri

See the agenda for more details.