thetestspecimen

Crowdfunding – What Is It and Are Sites Like Kickstarter Safe?

Tue, 22 Mar 2016 00:00:00 GMT

Crowdfunding is an exciting way of purchasing cutting edge design and technology, whilst also getting hands on in the development of some interesting and unusual ideas that typical investment routes may not favour or produce.

The question is, whether crowdfunding is an exciting and enjoyable journey, or a black hole of lost money and bad experiences?

In my experience, if you know what to expect before you jump in, it can be exciting and worthwhile, and you will end up with some unique and exclusive items!

At the time of writing I have backed 20 separate projects on Kickstarter (My backed projects) since I opened my account in Aug 2014. In that time I have backed projects ranging in value from £14 up to $495. This gives me a reasonable view of the ins and outs of the process.

In this article I have outlined my experiences and attempted to point out some of the pitfalls and bad assumptions people tend to make the first time they use crowdfunding.

I hope this article will help you avoid the common mistakes and get on with enjoying the rewards!

What is Crowdfunding?

Typically when you have a great idea, but not enough cash to make it happen, you need to get yourself an investor. Traditionally this meant pitching your idea to some rich guys or gals (think Dragons Den) who then decided whether it was feasible or not as an investment. Your idea may be a great idea, but it might have niche or limited appeal, thus limiting overall value to an investor. It could also be something that traditional investors don't have the foresight to appreciate, like cutting edge technology.

This is where crowdfunding comes in.

Crowdfunding allows you to generate the money you need, but rather than one or two investors investing a large amount of money, it involves a large amount of investors investing a small amount of money. Same outcome, different method. This has a few advantages:

If enough people are willing to stump up money to make your idea a reality, then maybe it IS a good idea
You gain the money you need to make the idea a reality
You have a large community to gain ideas and feedback from
You have your first batch of customers already (assuming you are producing something)

Where can I go to get involved?

If you want to get involved in crowdfunding there are two main places that you should probably start with. The first is called Kickstarter:

...and the second is called Indiegogo:

There are probably more crowdfunding websites out there, but these two have both been around a while and tend to dominate. In terms of my experience with crowdfunding, I have only actively used Kickstarter, so all of my comments in this article are based on my experiences there. Although the basic principles and ideas are the same on both sites.

So how does it work, and what do I get in return?

Basically, the way it works is that the creator of the product provides information on what they intend to create or provide to you. They also state the minimum total amount of money they need to allow the project to become a reality.

The creator also sets up the "rewards" they will provide if you "pledge" a certain amount of money (you could effectively read "reward" as "product" and "pledge" as "give them"). It should be noted that both the reward and the amount it will cost is set by the creator, not you, and is typically representative of the items value.

Once the campaign is started by the creator people are allowed to pledge money through the "pledge levels" (see the next section for details) for a set amount of time (usually a month). Over this time, the creator must get enough people to pledge money to support their product to ensure they achieve the monetary target they set at the beginning of their campaign.

If they don't reach the target, the campaign fails, nobody is charged any money, and the creator doesn't have to create the product. However, if the target is reached by the end of the campaign the money is taken from all the backers and given to the creator, who is then contractually obliged to (at least attempt) to produce or create what they set out in the campaign.

You can see an example below of some of the information provided. I have pixilated the names and pictures as this campaign is live at the moment, and as I know nothing about the product I would rather not unintentionally promote it!

5246 backers
the target was to raise $100,000 and so far they have $1,323,124 so it certain this campaign will be successful unless a large majority of the backers pull out at the last minute
there are 29 days left before the campaign closes. At that point the money is taken and the creation of the product begins

It should be noted that if you back a campaign you will not be charged any money until the campaign finishes. Up until the campaign finishes you can cancel your pledge and you will not be charged any money.

However, once the campaign finishes you will be charged for the product and cannot get a refund unless the creator agrees to refund your money or is proved to be fraudulent. At which point you would effectively need to take the appropriate legal action.

What are reward/pledge levels and how do they work?

The pledge/reward levels are set up by the creator to give you some choice, as you are only allowed to select a single pledge level. The different pledge levels are a combination of the following (I have used the Pebble Time campaign as an example due to its popularity and the fact it is now finished link):

Lower prices for backing the campaign early. Usually called "early bird" and have a limited availability. If the product is popular these places will disappear fast!
Different products. For example with the Pebble Time, there was the basic "Pebble Time" and the premium "Pebble Time Steel"
Multiples of the same product. As you are only allowed to select a single reward/pledge this means you cannot, for example, get multiples of the same product. To get around this creators will often make a pledge level that allows the purchase of multiples. For example, one of the pledge levels may say "2 Pebble Time Steel Watches"

You may have noticed in the above picture that the pledge prices actually have "or more" appended to the end of the price. What this effectively means is that the minimum you can pledge to receive the stated reward is, in the example above, $179, but you can pledge more if you wish.

As mentioned earlier, you can only select one pledge level, and you will only receive what is detailed in that pledge. If you select a pledge level, and then pledge twice the money in the hope you can have two, you will be disappointed...so be careful!

So how is this different from buying something from an online shop?

Well...this is a very important point, and often the main cause of people's complaint with the whole process. Crowdfunding is definitely not the same as buying something from Amazon or any other online retailer. The main reason for this is the time that you will have to wait to receive the product.

Each campaign will state a date they expect to deliver the product to your door. For example in the pebble campaign above you can see that the delivery date given was stated as "May 2015". For my particular reward on the Pebble Time campaign I had an expected delivery of "July 2015"

When did I receive the reward? Well although I can't remember the exact day they landed at my door, I received the tracking email 16th September 2015...so my guess would be around the 25th September 2015...and it doesn't end there. That is only when PART of the reward landed at my door. The "extra metal strap" mentioned above was delayed further due to production issues.

...so on a campaign that was run by a company that has used kickstarter previously to launch their original smartwatch (the original Pebble). They managed to provide an estimated date that was wrong by approximately one and a half months, and had further delays beyond that...and that was my specific delay, which will have been longer for some people.

My point is that:

if you look something up on a crowdfunding website and are expecting to be able to have it in a short space of time, crowdfunding is not for you
if you are willing to wait until the estimated delivery date but no longer, crowdfunding is not for you
if you want the item to replace something, and need it within a set timeframe, crowdfunding is not for you
I would also not advise getting something from crowdfunding as a present for someone as you may overshoot the date

Patience is essential, otherwise you are going to go crazy! I can only think of one campaign where I received my reward on time.

...and currently I have a project I have backed that had an estimated delivery of April 2015. I still don't have the reward, and don't expect to get it till maybe later this year!

Is it possible for me to lose my money?

The short answer is yes...and I have lost money (although not a lot).

Of the 20 projects I have backed so far I have lost my money on one of them. The project creator turned out to be a fraud, and although has promised to return my money has yet to do so (I'm not holding my breath!). In theory I could take this up with the authorities, as all project creators are bound by certain terms and conditions, but the monetary value was so small it wasn't worth my time. Although, some projects have been investigated as you can see here.

You may now be thinking that this is too risky? All I would say is that if you want to back something on a crowdfunding website common sense tends to prevail.

Check out the creators behind the project. Although they don't have to, it is typical that they are open and willing to expose their personal or company's identity. If not I would be suspicious
As with many things, if it sounds too good to be true it probably is

Any potentially risky items I have backed on kickstarter have been of low value. The higher value items were from established companies or traceable and easily checkable people. I would advise you to use the same approach. If you stick to this you won't go far wrong.

It is less convenient than an online shop, and potentially I could lose my money, so why would I bother?

Ok so now we get to the plus sides!

It is not an accident that I keep backing projects on kickstarter. I enjoy the process.

Here are the main reasons I use kickstarter:

I can find items that are simply not available anywhere else, either online or in my local shops. This is either because they are bespoke and unique ideas, or are pushing current technological boundaries
The price you get the product for is typically reduced compared to the retail price of the product after launch
You will be the first users of the new product
If the campaign is run correctly you will, at least to some degree, be able to influence the direction of the product. For example a current product I am backing has a unique colour scheme that was selected by the backers (and will only ever be available to the backers, never to retail customers)
In some cases you know that you have contributed to the start of a small company. Which can never be a bad thing!

Conclusion

To sum up, I would say that if you are interested in purchasing unique products, and are willing to be patient, crowdfunding can be an enjoyable journey. As long as you are sensible about the products you choose to back, and stick within the limits of what you can easily afford, you are unlikely to have many (if any) problems...and at the end of the day (or maybe at the end of the year!) you will receive a unique product that in at least a small way you helped to shape and create!

Nextbit Robin Smartphone – Kickstarter Edition (Unboxing and Photos)

Wed, 30 Mar 2016 01:00:00 GMT

In the latter part of last year I backed a Kickstarter project for a new phone called Robin, by a new startup called Nextbit. This article details the phone and accessories I received in the form of an unboxing video and photos, with some basic specs. I will provide a more thorough review once I have used the phone for a while.

You can check out the Kickstarter campaign page or their website here:

Kickstarter and/or Main Website

Nextbit Robin - Specification and Features

The phone is basically an android smartphone with a unique twist. The phone has a fairly standard 32GB of internal storage, but it also has 100GB of storage (for free) available in the cloud. Although I am not going to go into detail in this article I have provided a direct link to the specs of the phone here:

Specification

Nextbit Robin - Unboxing Video and Photos

I have made a video available on youtube which shows the unboxing of the phone and accessories. The video was shot at night, in a rush, on my birthday, so I apologise for the bad light. It should however show you what you need.

Since then I have taken some photos (in daylight) which should give you a bit more detail (see below for those).

Just for reference the photos and video detail the following:

Nextbit Robin Phone in the electric colorway/scheme - This colour scheme is exclusive to kickstarter, so is now unavailable, but you can still get the mint and midnight colorways detailed on the nextbit website
Bumps Fog Case
Scratches Frost Case
Quick Charger (EU version)
Glass screen protector
Nextbit T-shirt
Very special SIM tray (Gold)

I will provide more details on the phones software and general day to day usage once I have used the phone for a while, and if you have read this article and don't know what Kickstarter is you can check out my article which explains this here.

Please let me know if you want any further details/photos in the comments, and share if you like!

Unboxing video

Front of the Nextbit Robin

Back of the Nextbit Robin

Camera and flash

Nextbit

Very special gold SIM tray

Here you can see the glass screen protector installed. It is a good fit and fairly thin, you can just about see the lip it creates. It only covers the glass, not the top and bottom of the phone.

Top of the phone showing the headphone jack, also featuring the bumps fog case

Bottom of the phone showing the USB type C port and microphone, also featuring the fog bumps case

Nextbit Robin front with fog bumps case

Nextbit Robin back with fog bumps case

Nextbit Robin side showing the volume buttons and featuring the fog bumps case

Nextbit Robin side showing the power button (also a fingerprint scanner) and featuring fog bumps case

Scratches frost case from the back

Scratches fog case top

Scratches frost case bottom

Nextbit t-shirt front

Nextbit t-shirt back

Nextbit Robin – Software Review and 3rd Party Launchers (Nova)

Sun, 17 Apr 2016 01:00:00 GMT

The Nextbit Robin incorporates 100GB of free, automatically organised, cloud storage as part of the phone purchase. To integrate this functionality into the phone Nexbit has created a unique launcher which ships with the phone.

The question is: How good is the launcher, and if you don't like it is there an alternative without losing cloud functionality?

...So what is the answer?

Well to put it bluntly, the Nextbit Robin's launcher is somewhat lacking in areas that I wasn't expecting (more details below)...fortunately alternative launchers can be used and cloud functionality will work. The downside is that some of the design features integrated into the Nextbit Robin's native launcher do not transfer over to the 3rd party launcher. This means app management in terms of cloud storage can become cumbersome.

It seems, for the moment at least, whether you stick with the native launcher or move to a third party launcher you will have to compromise at some level.

Checkout the video below which runs through the basic launcher functionality, and then goes on to discuss how Nova launcher functions on the Robin. If you would rather not watch the video I have included some screenshots in the article below to cover the same areas:

What is the Nextbit Robin?

If you want a brief overview of what the Nextbit Robin is about you can check out my previous article here Nextbit Robin. You can find out more about Kickstarter, which is the platform the Nextbit Robin was launched on here Kickstarter.

How exactly is this a cloud phone?

The robin comes with 32GB of onboard storage, but your apps and photos are automatically uploaded to the 100GB of cloud storage (I believe this should also be possible with videos in the near future).

Cloud Storage and Archived Apps

The basic idea is that the phone manages your photos and apps for you between the cloud and local storage, so you don't have to worry about running out of space. You could break this down into two areas:

Photos

I like how the photos are handled, it is quite clever. You take a photo, the phone uploads the full resolution photo to the cloud, leaving you with a downsized version (I believe 1080p, i.e. screen resolution) on your phone locally. If you think about it, this is perfect for on phone browsing. Should you then want to share an image the smart storage will automatically send the full res version when you attach it to an email or share.

Apps

Applications are handled a little differently. Should you start to run out of storage locally the phone will start to delete apps off your phone (don't panic they are still in the cloud!). It starts with the least used apps and moves up the list from there. The icon for the app stays on the phone, but is greyed. Should you want the app back, just tap the grey icon and it comes back with all the app data too! As if you never uninstalled it, so no need to sign into the app again.

Restoring Snapchat

What about the native launcher?

The "Nextbit" launcher that comes with the phone is somewhat lacking in my opinion. In many ways it is a few backward steps in terms of ease of use and functionality considering how far the android ecosystem has now come. The best way I can think of describing the implementation of the launcher is "iphone like". If that's your thing great, but I think it fails in terms of functionality.

Consider what this phone is designed for. It is designed with 32GB of onboard storage, with an extra 100GB in the cloud. Great! So this would imply that the people it is aimed at are heavy users of storage. This could of course be in terms of videos (not yet supported) or photos only, but it is likely that it will also involve a reasonable number of apps.

If this launcher was designed with use of a large amount of apps in mind I am struggling to see how this went through development without an issue. Then there is the implementation of the widgets... Here are some of my issues:

There is no app draw, so all apps live on the home screens
There is a sort of app draw available using the purple button which sticks to each homescreen in the bottom right, but it is just a vertically scrolling list of alphabetically sorted apps, and no way to search it quickly!
As I have mentioned, the homescreen is just for apps, so where are the widgets? Well if you pinch the screen you can see (and add to) the widgets. Basically they are not visible unless you pinch gesture. Really! I thought the whole point of widgets was to serve up information from within apps so it is visible easily ON the homescreens not BEHIND them?!

In terms of other functionality of the launcher:

You swipe down on an app icon to "pin" it. This means the app will never be removed off the phone even if you run out of space
There are three separate lists of apps under the purple button, which show you archived, pinned and all apps, so you know what the status of each app is
You can bulk restore or completely uninstall archived apps (see screenshots below)
You can bulk unpin apps (see screenshots below)

Of the three issues I have mentioned above, it is number three that really baffles me. I can only assume that Nextbit came up with a way of integrating their cloud functionality by having interactive apps on the home screens (to allow pinning of apps by down-swiping an app icon, see the video for a demonstration) and then couldn't figure out what to do with the widgets.

Purple button menu and all apps and archived apps "draw"

Restoration message, pinned apps "draw" and widgets screen

Can I use a different launcher such as Nova Launcher, and what happens to the cloud functions when I do?

The simple answer is: yes you can, and the cloud functions will work (with a few caveats).

Nova Launcher App Drawer

Nova launcher works as expected, but the important question is how the cloud features work.

As you can see from the image above, the application icons do grey out when they are offloaded to the cloud, which is great. I currently have my nova launcher app draw setup with tabs across the top. What happens when the phone archives an app, is that it moves from a specific tab, for example "Games", to the "Apps" tab. When you restore the app it goes back to correct tab. Which I suspect is more to do with Novas settings than the phones.

If you want to restore an app you just tap the greyed icon and it will be brought back down to your phone. You won't see the nice reloading animation that you get with the Nextbit launcher (see the images above with snapchat), but it performs the same function at the end of the day.

Another problem with restoring apps from Nova Launcher is that you can only do them one by one (i.e. you can't restore all or queue them)...which is tedious if you have many apps to restore.

Another problem occurs when you want to pin apps. You would usually achieve this by swiping down on the icons on the homescreen to pin an app. As far as I can tell this doesn't work in Nova Launcher, and there seems to be no way to do this from any settings menu. Your only option is to go into the Nextbit Launcher, pin the apps you want, and then go back to Nova Launcher.

You also have no way of seeing which apps are pinned without switching launcher.

That is about it, but it amounts to losing quite a few convenient features of the Nextbit Launcher that make the extra cloud features easy to use.

Anything else I should be careful of?

Yes.

One thing I noticed when testing out the cloud storage is that if you start to run out of local storage space there are some applications that the phone started to automatically archive that I didn't want it to, but hadn't thought of. Basically, any apps that you use all the time, but never open. Examples of this in my case are:

Any license checker apps (Nova launcher prime in my case)
Any live wallpaper apps (Chrooma in my case)

Because the phone is ranking the apps in terms of usage, the above types of apps that are required but never opened are seen as the least used apps, and therefore are the first to be archived. I would recommend going through your apps and pinning the apps that fall into this category so you don't, for example, lose your wallpaper suddenly as happened to me!

In conclusion

The phone itself has a unique design and is well made. The implementation of the cloud storage has potential, but at the moment it is a little lacking in features (video archiving) and I think the launcher is a step in the wrong direction. If the development continues I can see the potential, so hopefully the next iteration will be one to watch out for.

Would I recommend the phone? Yes I would, but because of the style of the phone, build quality, barebones android implementation, unlocked network and bootloader; not necessarily for the cloud storage feature.

My Android App Development Journey (So Far), and Some Guidance for Others

Sun, 12 Aug 2018 01:00:00 GMT

About a year and a half ago I decided that I wanted to try something new. I love learning new skills and I wanted something to focus my mind and give myself a new challenge. I decided the "thing" would be to try and learn to computer program, but where to start? Well I did start and (while I can still remember) I wanted to write down some of the experiences, frustrations and successes of the last year and half, so that I can look back at them, but also to try and help others that might want to start the same journey.

This is my journey from not knowing how to code, to publishing a fully working android application on the Google play store (more info about the app at the website here). I hope you find it useful.

Why computer programming?

There are a couple of reasons:

I like technical subjects

I'm an engineer by trade, and this means that I'm generally quite good at technical type subjects that involve logic, problem solving and patience. I would say computer programming falls into this category.

Accessibility

The other main reason is accessibility. What I mean by this is that, typically, at least in western countries, a computer is already available to most people. The information available on the internet is also extensive, especially with regard to computer programming.

Overall this means that with little to no outlay you can begin, and likely complete, what you set out to do.

Why did you pick Android?

This was initially one of the sticking points. There are hundreds, if not thousands of programming languages to choose from, and there are new ones coming out all the time. So how do you pick?

My criteria

I decided the best way to approach it was to set some high level goals:

I must be able to produce something complete by the end (i.e. a complete app or program that I can actually use, not some abstract code)
The tools to build the application or program to completion must be free
It needs to be mainstream enough that I will have enough resources available to successfully learn. Otherwise frustration or boredom (or both) will likely set in
It should be a good representation of current modern programming languages so that I can potentially move forward with other languages at a later date
It should be as self contained as possible. I wanted to avoid any confusion that will likely come from having to get my head around which tools are required to accomplish the job. I wanted to concentrate on learning to code.

Using the above criteria I did some research...

Thinning it out

The first port of call was potentially web development (i.e. making a website). I threw this out quite quickly as the ecosystem is very complex. It would require the knowledge of HTML, css, php, javascript (and more...) and the tools to deal with these are even more numerous and fast evolving. I needed something more contained...

It needed to be a program or app. I could therefore pick a platform:

iOS (requires monetary outlay as I have no apple devices, and an annual fee to release an app)
Mac (requires outlay for a device)
Windows (maybe, but the tools I would use to make it were not clear enough, and C++ is maybe not the best beginner language)
Linux (not familiar enough with the platform)

The decision

...which left Android:

I have many android devices already, so I'm familiar with the platform usage and will have no hardware outlay
Android Studio (the program used to make android apps) is free and self contained. You only need to use that one program, and you can output a complete working app.
It uses only two languages. Java and XML, which keeps it simple. Note: you can also use Kotlin, a new language, but it wasn't available when I started out.
Java is a good representation of a modern language, and one of the most utilised programming languages in the world
There are extensive tutorial videos and websites on how to code in Java AND how to make an Android application. I should therefore be set in terms of learning material

Now I had that out of the way, how to start?

Where did you begin?

I think this is a good point at which to state my level of knowledge with regard to computer programming before I started.

Mainly through my experience as an Engineer I had some basic knowledge of what a for loop and if statement were through usage within Microsoft Excel VBA macros (don't worry if you have no idea what that is, it doesn't really matter). I didn't write the macros, rather adjusted macros that were automatically generated by excel. We are talking really basic stuff...

If you know what things like strings and ints are then you are way ahead of where I was. If not, don't worry, neither did I!

Learning the basics

With the above in mind I thought the best place to start would be with Java. Java is the main thing I would be using while writing an android application, so this seemed sensible. I needed to start from the basics. Now I know this was a good choice, as it made following the android tutorials that much easier later on.

I started searching for places I could start to learn Java. Probably the best place to start is with online videos. There are many on youtube, but you can also find some great courses on sites like udemy.

I spent a long time jumping between different courses (some are terrible and hard to follow) and youtube can also be hit and miss, but I am going to make a solid recommendation.

Recommendation

There is an Australian developer who produces online tutorials called Tim Buchalka. His videos are easily the best I have seen, and as a plus, he has video courses on both Java AND Android development (and Kotlin now too). Both are excellent. They are not free, so this may not be an option for some, which is fine, but if you can spend a small amount then you won't regret it (his Complete Java Development Course is currently EUR 9.99 on udemy just to give you and idea of price). [I also want to make it absolutely clear that I am not affiliated with either Udemy or Tim Buchalka, and receive no kickback from either.

Practise makes perfect

Whether you decide to go with the video suggestion above or with text books, there is one thing that you absolutely must do, and that is actually write the code yourself.

You can sit for hours reading and watching videos, but it will be no use if you don't actually put it into practice.

You can sit for hours reading and watching videos, but it will be no use if you don't actually put it into practice. I cannot stress this enough, it is essential!

Which software did you use?

For Android it is easy, you use Android Studio. You can download it for free directly from Google. You will literally never have to use any other software if you don't want to.

Other useful software

It is however useful when you are learning Java (or Kotlin) to have an interface that is just for Java and not cluttered by all the other things that Android Studio includes (i.e. keep it simple). There are two programs I ended coming back to again and again:

IntelliJ

IntelliJ is excellent for writing Java (and Kotlin) code as it is designed for Java development, it is also free to use. The second advantage of IntelliJ is that Android Studio is made by JetBrains, the same company who make IntelliJ, so the interface is very similar. Learning with IntelliJ will help you later with Android Studio.

As an aside, the programming language Kotlin was invented by JetBrains, and is now an officially supported language of Android development. Interestingly, Android Studio will actually convert Java code into Kotlin code for you if you want, as they are similar languages.

Sublime Text

Sublime Text is almost like an advanced notepad, which is specifically designed for coding. It doesn't look very impressive at first sight, but you will keep coming back to it because it is light and fast. Sublime text has a weird payment model. You can essentially download it from their website, and use it for free forever. However, officially you should buy a licence for "continued use". I would say that if you end up using it extensively then that is fair enough, the developers certainly deserve to get paid!

How did you progress and what were some challenges you faced?

I will start by saying that programming is a rewarding, but at the same time frustrating process. It has definitely been worth the time, and you will, and should, never stop learning. For want of a better phrase, app development is a bottomless pit of things to learn and master.

First Steps

As I have mentioned above I started by learning Java. This was very enjoyable, and it's really fun seeing your first lines of code run. The very first program being "Hello World", which seems to be a tradition for programmers as the first code output you ever make.

app development is a bottomless pit of things to learn and master

Then I moved on to trying to understand some concepts of Object Oriented Programming (OOP). This is worth looking at as it underpins a lot of modern languages, and certainly Java. This can take a while to get your head around but is not too bad at a basic level, and ultimately it will help you take advantage of the powerful features the language has to offer.

Android Studio

Then I decided I was ready to jump into android development. Before you even begin you need to get your head around Android Studio. It is an excellent program, but you will need to learn where everything is, and what the correct naming convention is for files etc. The best way to do this is to follow a video tutorial for a basic app, and then do it yourself in android studio.

Learn from others

In addition it is a really good idea to take a moment and see how other developers layout their android projects. This you can do by looking at the structure on some GitHub projects (check this article for some links to some GitHub Projects). It allows you to see what convention developers tend to follow:

class names
package names
method names
how long are methods typically
where do they write notes, and what do the notes contain
where are resources kept (like pictures etc.)
what libraries are used a lot
how do they write their code (code style)

You can certainly learn a lot from other people's successful projects. This will also help you avoid some pitfalls that you may otherwise have fallen into.

Progression

At this point I have managed to get android studio up and running, I know some Java, and I can knock together a very, very basic app, and run a test version on my phone. At this point I started the "main" app I had in mind.

Starting with the initial screen I began to build. In my mind I had an image of what I wanted to achieve, but due to lack of experience you come upon a couple of problems:

I had no way to plan ahead properly as I didn't have the knowledge to know what makes the most sense to proceed with first, and I didn't know how to properly structure my code to make life easier later
I realised that some of the ideas in my head, although on the face of it seemed simple, were not necessarily that simple to implement. (Sometimes it can be the reverse! You think it will be hard, and it takes two lines of code, but this is rare.)

The outcome of this is that you will write code, learn, and then re-write the code later when you realise the mistakes you have made, or the inefficiencies in your code. This makes the whole process longer, but it is the best way to learn.

For example, the first problem you will likely hit is that you will write all your code in one big block. Then later you will realise this is terrible as you can't follow your own code, and you will break it into smaller sections or separate files, as it should be. Seeing the problems that large blocks of code causes teaches you why you shouldn't do it. This process repeats multiple times for various different settings

I'm a coding god!

This is the next stage of the journey. There will come a point when you think "Aha! I get it, this is not so bad now. I think maybe I'm a natural!". It runs well on your phone, it looks like you want (almost)...

Aha! I get it, this is not so bad now. I think maybe I'm a natural!

Not long after, it will dawn on you that you have only scratched the surface. Yes, you can code, but due to your new knowledge you will have a better view of the whole picture, and it is vast. Here is a small selection of the things that popped into view once I stepped back a little and reviewed what I had achieved so far:

Is your code backed up on something like Git (What is Git??!!)
Have you properly tested your code with things like unit tests or similar (What?!!)
Do your resources load efficiently, fast and not on the main thread (That sounds complicated!)
Do you have any memory leaks (How do I even look for that?)
Is your code structured in a way that is easily expandable and readable (to me yes...but others maybe not)
Does it run without crashing on just your phone and android version, or ALL phones and android versions (hmmm...)
Is your code using the correct method of implementation for a particular task, or is it just a hack of inefficient methods because that's all you know at the moment (...more code review then)

Ok, I'm not a coding god...

My point is that you will never know everything, the subject is huge. If you think you do then you are not trying hard enough, or you are doing it wrong! As I now know, even large companies like Google with their thousands of well paid software engineers do not have perfect bug free code. I know this because I have personally had to work around various bugs in Google's code while making this app.

even large companies like Google with their thousands of well paid software engineers do not have perfect bug free code

A really good resource for any questions you have about android, java or any other coding is StackOverflow forum. I literally wouldn't have a working app without that forum. It is a goldmine of information.

Then one day I asked myself...

Ah such a simple question, but what a can of worms.

The thing is by this point I was fairly committed to the app, so I wanted to do it properly. I wanted to be able to record my data in the app, have it sync across devices, and be backed up.

You can use various services to do this for you that will make the interface simpler...but you will pay eventually, as it is initially free to entice you in, and then the costs quickly ramp up with usage. You will also be locked into their system. I wanted to avoid that. Which left me with one option. My own server.

Getting a server

I signed up for a Linode VPS (which I thoroughly recommend) for $5 a month and got cracking. I managed to build a Linux Ubuntu server. This also had the advantage of allowing me to setup my own emails, and an associated website so I could explain how the app worked (yet more work I hadn't planned for). I then created a REST API using Django Rest Framework (essentially a communication protocol between the app and the server).

Again, this took time to research and implement. I had done none of this before, and in the process I had to learn a new language called Python (not as bad as it sounds once you know a language like Java to some degree), and how to administer and secure a webserver (this was a little involved, but worth it).

Then I had to get the webserver to communicate with the app! To do this correctly was the biggest challenge to date.

Another recommendation

I will make another recommendation here. When you are dealing with a REST framework you need a program to test it with. I won't go into the detail here, but one of the most popular is called Postman. However, I found the interface a little clunky, and I found an alternative which is (in my opinion) much better: Insomnia. Give both a try if you like.

One last hurdle

As I went through the process of making the app, I would often come up with new ideas or changes I wanted to make, but at some point you have to actually release the app!

I therefore decided to draw a line in the sand, make sure the app functioned smoothly and release....then I noticed GDPR.

If you don't know what GDPR is, then I will assume you are a hermit living in a cave, but I'll humour you. It is basically legislation implemented by the EU that affects any "thing" (business, website, app, service etc.) that may be used by an EU citizen in terms of privacy and rights to personal information. Overall this is a good thing for all our internet related futures, but it meant that I had to implement various changes to ensure I was in compliance, as the criteria is very strict.

What didn't help was that the efforts of large companies such as Google were a little half-hearted in relation to helping out developers on their platform adhere to the legislation.

This set me back a good month, if not more. Absolute pain in the ass. Anyway...

Finishing up

I finally got to the point of release. Here it is if you want to give it a spin, or search for "Wanderfile" on the play store:

Wanderfile Website (Explains how the app works)

It is a really excellent feeling, but it doesn't end there...you need to keep it up to date and promote the app. You are competing with thousands of other apps, and to even get a little bit noticed is a struggle. Don't expect 1000 downloads on your first day (or week, or month for that matter).

I have no idea if the app I made will succeed or not. It certainly won't be the next Facebook, but that was never the intention.

What I achieved

I created what I wanted to use, but more importantly, I learnt a huge amount. I can now, at least to some degree:

Administer a Linux Server (Ubuntu)
Write an Android application
Write a website from scratch using html, css, php and javascript
I know how to setup an email server for personalised emails including web access secured by second factor authentication
Can write in Java and Python programming languages, although less so in Python
Have dealt with various frameworks for the website and app (e.g. Django Rest Framework)

I would never have thought I would get this far, and certainly didn't expect to get involved in so many areas. It was, in the end, consistent hard work in my spare time. Occasionally it felt like a chore, but generally it was a series of little rewards, with a large reward at the end.

The worst that could happen is you will learn something.

If you think you might enjoy coding I would encourage you to give it a look, and try. It is free, accessible and very rewarding. Plus you will learn things that are applicable to a whole host of fields. The worst that could happen is you will learn something.

Would you change anything looking back?

Nothing. Honestly. I messed up plenty, but it is all part of the learning process. Now I know things I didn't even have the first clue about before.

What are you going to do now?

Now I will do my best to keep the app up to date and add new features. I will blog a little here and there, and I will try to think of the next app I could potentially make.

Next challenge basically!

This article is already too long, but I could write forever detailing exactly what I did and didn't do. With that in mind, if you have any questions that you would like to ask either send me an email and ping me a message on twitter and I'll be happy to give you some insight.

Linux – Should I Switch from Windows to Linux (Ubuntu, Fedora) or Not?

Wed, 15 Aug 2018 01:00:00 GMT

Linux still has the reputation of being a geeky alternative to Windows and Mac that snotty little know-it-alls use to make everybody else feel inferior and stupid. I would say this reputation is wrongly attributed, and in its current form Linux provides a real alternative to Windows for most people. I will do my best to explain why I think this is so...

The question is: is Linux a good replacement for everyday computing, and should you be using it instead of Windows?

What do I use and prefer?

Personally I use both Windows (still Windows 7 actually) and Linux. On a day to day basis I use Linux, with windows usage coming in at maybe once a month.

I much prefer using Linux for a variety of reasons I will get into, but that doesn't mean I think it will suit everyone.

If I decide to use Linux, which distribution should I use?

This is the main problem that Linux has at the moment.

Linux is the type of operating system, but there are many different versions of Linux:

Ubuntu
Fedora
Manjaro
Arch
OpenSUSE
Mint
Gentoo
Mandriva

...and there are more, a lot more.

Basically every version of Linux uses the same core (officially called a Kernel), so what you are really deciding between is the user interface, and how well it is maintained as a system. The "core" functionality is the same.

I will therefore make this very simple for you. If you have never used a Linux distribution before you either use Ubuntu or Fedora.

These two distributions are the biggest, have the best support, and will therefore make your life easier. Please don't assume that because they will make your life easier they are missing attributes that some other distributions have, as this is not the case.

Ubuntu is easily the most widely used, but personally I have had a better experience with Fedora (I use Fedora on a daily basis, as does the original creator of Linux Linus Torvalds). You can't really go very wrong with either.

If you eventually decide you like using Linux you can look into the other variants, but chances are you will not need to unless you have very specific needs.

What are the advantages of using Linux?

It is absolutely free
The software you install is predominantly free
It is fast, and doesn't have the same amount of annoying fluff as Windows
The user interface is simple and easy to use
It is by nature much more secure, and much less prone to viruses
It is very easy to update
It can be portable (usb drive, external ssd/hdd). You can't do that on Windows!

The other thing to bear in mind is that Linux as an operating system is not small in terms of market share. There are many, many things that use Linux:

All of the top 500 supercomputers in the world use Linux (i.e. the fastest computers in the world)
66% of web servers world wide use Linux
75% of all smartphones and tablets use Linux (this is because Android is based on Linux too)

The reason I mention the above is that it would be wrong to assume that Linux is less mature than Windows. Linux is a very advanced and constantly updated system, even Microsoft work on improving Linux, that is how important it is as a system.

What are the disadvantages of using Linux?

If something goes wrong, there are less resources available on the internet to help you fix it
If something goes wrong, it is usually more difficult and involved to fix it
You will likely have to use the commandline at some point (this is not really a negative as it is extremely easy, and I actually prefer it)
You do not have access to the same programs as Windows has, but there are typically very good alternatives
There will be a learning curve initially

I mentioned in the previous section about the extensive use of Linux in various systems, but if you only consider the desktop environment that consumers use, then the story of usage is a little different:

Windows market share: 88.5%
Linux market share: 2%

This translates into, less information being available online (less help forums etc.), and less software being made available for the system. The reality in day to day use is not as significant as the numbers would suggest, as the Linux community online is knowledgeable and active. That 2% are generally active contributors, and know what they are talking about.

Who shouldn't be using Linux?

I think this is potentially an easier question to answer than who "should" be.

If you are someone who absolutely has to use proprietary software then Linux is not for you

If you are someone who absolutely has to use proprietary software (usually due to work) such as Adobe Photoshop, Adobe Premiere, Microsoft Office Suite, specific CAD software such as AutoCAD etc. then Linux is not for you. As currently you will only find software like that on either Windows or Mac.

This is the only reason that I keep a version of Windows to hand, sometimes I still need software that is only available on Windows. This is however becoming less and less the case.

Who should be using Linux?

I think most people will find that using something like Fedora is actually a breeze on a day to day basis, so I would probably say that if you don't have to be tied down to specific software, or you only have basic needs, then it will suit you down to the ground.

I would even go as far as to say that I think my parents would find it easier to use than windows

I would even go as far as to say that I think my parents (in this case representing a generation with not so great computer skills) would find it easier to use than windows, as it is more intuitive. There are less menus and settings to confuse them, and finding things is much simpler.

What if I need programs like Microsoft Office?

If you are not tied rigidly to any specific software or suites, you can switch them out for very capable alternatives. Just as an example you can swap out as follows (all free of course):

Microsoft Office --> Libre Office (or something like Google Docs, Sheets etc in the cloud)
Adobe Premiere --> KdenLive
Adobe Photoshop --> GIMP
Adobe Illustrator --> InkScape
CAD --> FreeCAD
Microsoft Outlook --> Evolution Mail

These are just examples, but there are many more, and they are very comprehensive programs (don't view them as second class rubbish, they are not).

What about the commandline, that sounds scary?

The commandline in Linux is not scary at all. It is much easier to use than having to delve into the travesty that is the settings menu structure of Windows. Let's take a couple of examples:

I want to install a program

I want to keep this simple on both platforms, so it is a fair comparison. Filezilla is an FTP program (could be anything it doesn't matter) available for free on both Windows and Linux, so...

Windows

Go to google in a browser
Find filezilla website
find download page and correct version
download file
find where downloaded file has saved to
double click file to install
go through the install guide (being careful not to accidentally install the guff that usually comes with installers of free programs [don't think filezilla do this, but it is way too common...])
Open program

Fedora

Open terminal / commandline
type "sudo dnf install filezilla" (the first three words never change, so it's easy to remember)
type password (this is where the security of Linux becomes clear, you cannot install a program without a password, sounds like a pain, but it is MUCH more secure)
Accept install by typing "y" and pressing "enter"
Open program

My point is with the above example that you have to go all over the place to install something on Windows, but you never have to leave the commandline in Fedora, not even once.

This also illustrates how it would be simpler for the older generation. Installing a program on Windows involves downloading the exe file, but I have seen many times with older people that they have no clue where it goes once it's downloaded! Bit of a problem if you ask me. Fedora does not have this problem, it just works.

Other commands

I'll assume you are familiar with the process of Windows at this point, so I will give a couple more examples of how easy things are with the commandline for everyday tasks. Try to remember that all this becomes second nature after a while:

Update operating system: "sudo dnf update"
Uninstall filezilla (or other program): "sudo dnf remove filezilla"
Open filezilla: "filezilla" (you can of course go and click on the icon too)

You can do anything from the commandline, which makes it very powerful. There is plenty of guidance on the internet too.

Conclusion

Is everybody suddenly going to jump on Linux, no I doubt it.

When you are used to a system it can be very difficult to switch over to something new. However, I think that in general peoples usage of traditional computers is changing. People are moving towards more mobile, more cloud based and more secure systems that are a breeze to maintain. This has seen the rise of things like the Chromebook, which auto-updates, is cheap, fast and gets your day to day jobs done easily (it is also based on Linux by the way!) .

My point is that people are becoming more open to alternatives to Windows in general, and I think this could be good for Linux as it really is an excellent alternative.

At the very least I would give it a shot. It is free after all. You don't even have to get rid of windows to try it because you can:

Install it on VirtualBox in Windows (this is easy there are plenty of tutorials on YouTube)
Burn it to a DVD and run a "Live" version from the DVD. This basically means you can use the operating system to see how it works, but once you log off all data is lost (great for trying it out though!)
Write it to an external drive and delete it later if you don't like it

It is much more flexible than Windows in this regard, so you can try it without losing Windows.

How to Make Broadcom WiFi Modules Work in Fedora

Sun, 02 Sep 2018 01:00:00 GMT

Due to the fact that broadcom use proprietary drivers for their wireless modules it is not usually possible for them to work out of the box with Fedora. This tutorial is aimed at helping you to get the adapter working.

Notes and Pre-requisites

First I want to mention a couple of points:

to complete this tutorial you will need an internet connection. I know that sounds a little silly, but there is no other way. You therefore need a temporary wifi adaptor that works in Fedora already, or a hardwired connection
this process has been tested on Fedora 26, 27 and 28, and I know it to be working for those systems
the specific wifi adapter I have is BCM4352 (802.11ac Wireless Network Adapter (rev 03)). Although this process should work for all (or most) broadcom adaptors I can't guarantee it
UEFI secure boot must be disabled in BIOS

Let's get started

The process itself can be quite simple, but I am aware of some pitfalls that you may come up against in some cases. I know how to get around some of these. With this in mind I will try to highlight potential problems and their fixes as we go through.

The simple way

You should first make sure you are up to date

sudo dnf update

Then add the rpmfusion repositories

sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

Get the broadcom drivers:

sudo dnf install kmod-wl akmods akmod-wl

Check if you have kernel-devel installed. It should be, but if not this will install it for you.

sudo dnf install kernel-devel

Now reboot / restart your system

Once the system has booted and you have logged in again we need to check if the driver "broadcom-wl" has installed. Obviously the the actual driver version might be slightly different to the output below, but you should get a similar response

[testspecimen@localhost ~]$ rpm -qa | grep broadcom-wl
broadcom-wl-6.30.223.271-1.fc22.noarch

Run modprobe

sudo modprobe wl

Check that akmods can be run

[testspecimen@localhost ~]$ sudo akmods

Checking kmods exist for 4.17.19-200.fc28.x86_64           [  OK  ]

There is a chance that you will get a warning that "files needed for building modules against kernel could not be found" and "Is the correct kernel devel package installed?". If that is the case don't worry, just continue.

We then need to check that the kernel-devel package in the system is the same as the kernel-devel package installed with the $(uname -r) kernel of your system. First check the system:

[testspecimen@localhost ~]$ ls /usr/src/kernels/
4.17.19-200.fc28.x86_64

then the $(uname -r)

[testspecimen@localhost ~]$ uname -r
4.17.19-200.fc28.x86_64

If they are the same you can skip to the "Load the "wl" module" part just after the system restart in red below, otherwise you need to do the following

You now need to install the correct kernel devel for your system. This is the version that was output when you ran the "uname -r" command. Go to the following website:

Kernel Devel RPM files

You then need to find the correct rpm file for your system for example in my case it would be:

kernel-devel-4.17.19-200.fc28.x86_64.rpm

...and download the file.

Once downloaded (I am assuming you will download the file to the Downloads folder) run the following commands replacing the rpm file name with the correct one for your system that you just downloaded

[testspecimen@localhost ~]$ cd ~/Downloads
[testspecimen@localhost Downloads]$ sudo rpm -Uvh --force kernel-devel-4.17.19-200.fc28.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:kernel-devel-4.17.19-200.fc28 ################################# [50%]
Cleaning up / removing...
2:kernel-devel-4.17.19-200.fc28 ################################# [100%]

Run akmods again

[testspecimen@localhost ~]$ sudo akmods
Checking kmods exist for 4.17.19-200.fc28.x86_64 [ OK ]
Building and installing wl-kmod [ OK ]

Reboot / restart the system

Load the "wl" module

sudo modprobe wl

Check "wl" is loaded

[testspecimen@localhost ~]$ lsmod | grep wl
wl                   6463488  0
cfg80211              770048  3 wl,rtlwifi,mac80211

Check that module "wl" is used as a wifi adapter. Please note that the terminal will output a lot of different components like the one below, so you will have to search for the wireless controller specifically.

[testspecimen@localhost ~]$ lspci -v
05:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4352 802.11ac Wireless Network Adapter (rev 03)
    Subsystem: ASUSTeK Computer Inc. Device 855c
    Flags: bus master, fast devsel, latency 0, IRQ 16
    Memory at dfa00000 (64-bit, non-prefetchable) [size=32K]
    Memory at df800000 (64-bit, non-prefetchable) [size=2M]
    Capabilities: <access denied>
    Kernel driver in use: wl
    Kernel modules: bcma, wl

At this point unplug the temporary wifi card you are using or unplug the hardwired connection you are using. Then we can get a list of networks up

[testspecimen@localhost ~]$ nmcli dev wifi list
IN-USE  SSID            MODE   CHAN  RATE        SIGNAL  BARS  SECURITY  
        OnePlus5        Infra  11    130 Mbit/s  95      ▂▄▆█  WPA2        
        COSMOTE-F0CEBB  Infra  6     270 Mbit/s  57      ▂▄▆_  WPA1 WPA2    
        COSMOTE-0EF1CE  Infra  11    270 Mbit/s  35      ▂▄__  WPA1 WPA2 
        conn-x71aa38    Infra  4     135 Mbit/s  34      ▂▄__  WPA1

Then you can connect to the wifi network of your choice

[testspecimen@localhost ~]$ nmcli dev wifi con "your_ssid" password "your_ssid_password"

That should be it!

Let me know if it worked for you, and if so which driver it worked for so I can create a working list for other people to reference.

Credits

After many hours trawling through the internet I found the method detail above worked for me. The information in this article predominantly comes from one place:

Source

Cryptocurrency Beginners Guide – What Are They, Should I Get Some, and How Does It Work?

Wed, 19 Sep 2018 01:00:00 GMT

The aim of this article is to get you up to speed on the basics of cryptocurrencies. I will go over some of the main terminology, and try to give you a good idea of how to get hold of some, and how to store it properly. Pointing out the differences between standard "fiat" money like dollars / euro / pounds and cryptocurrencies will also feature.

Once you get to the end of the article I hope you will understand what all the fuss is about, and have a better idea of why it is much more than a speculative money making investment.

Cryptocurrencies have been all over the news in recent times, and there are a lot of cryptocurrencies on offer (literally thousands), including the ubiquitous Bitcoin.

Unfortunately, the main focus has been on short term speculative gains, rather than the core technology and potential advantages that cryptocurrencies offer.

Understandably, this may put some people off. However, I would encourage you to look beyond the current hype, as there are a lot of revolutionary ideas in cryptocurrency. It could dramatically change the way the world operates in the not so distant future.

It really is a world worth exploring.

What I won't be talking about...

A lot of articles about cryptocurrencies focus on one subject:

Which coin is going to make the largest gain in the near future, or make me a millionaire in a few years?

Well if you came here for an answer to that you are in the wrong place. Just wanted to get that out of the way early so I don't waste anyone's time...

What exactly is a cryptocurrency?

You could say that it is "digital money", but that is a meaningless statement. The money you currently use dollars / euros / pounds is already digital. You don't keep all your money in cash form do you? Most of it lives in a "digital" database organised by your bank.

So if we are already digital, what exactly is cryptocurrency? I have outlined below some of the key differentiating factors between normal "fiat" currency and cryptocurrencies in general:

Transactions are permanently stored and cannot be changed, ever, by anyone
The currency is decentralised. There is no single central authority such as a bank, government or person that controls the currency. It is distributed across many locations around the world.
Transactions are secure by default due to public-private key cryptography. You don't need to trust any intermediary (such as a bank) to make a transaction.
You don't need permission from anyone (government, bank etc.) to use them, as they are free to access for all
Transactions have no borders. You can send cryptocurrencies anywhere and to anyone with no limits or restrictions on geographical location or amounts.
Transactions are pseudonymous [this is not the same as being anonymous, although there are some cryptocurrencies that are anonymous]. You don't have a name on your account like a bank account, so people won't know who you are unless you tell them.

What this boils down to is this:

You can send money to someone, and receive money from someone else. All without having to deal with a bank, or even having a bank account. You can also have absolute confidence that once the money is sent or received it is permanent, verifiable and cannot be changed or reversed.

How easy are they to use?

It is a new way of doing things, so it is strange and confusing to try and understand at first. I will go over some of the terminology in the next section. In reality though, once set up, it is not a lot different from how you currently use your bank account:

You have an account number that you can give out to people so they can send you money (called a public key in cryptocurrencies).
With a bank account you confirm a transaction is real using your debit / credit card and pin (or card details online). With cyptocurrencies you have a private key (like a password) that you need to use to "sign" transactions. This can be simplified even further with hardware devices, but we will get to that later.

The main issue currently

The main issue at the moment is that you can't use cryptocurrencies in the real world everywhere, which limits their current usefulness.

What can you use cryptocurrencies for right now

You may be pleasantly surprised as to where you can already use cryptocurrencies.

Shopping

I would suggest taking a look at coinmap.org, which shows a heat-map of some of the places in the world where a form of cryptocurrency is accepted. There are even some big companies that will accept them such as Expedia, Newegg, Gyft etc.

More interestingly Shopify, which allows people to create online shops, has integrated Bitcoin payments into their online shopping cart. This means that any company, big or small, that wants to accept Bitcoin payments through their online shop can do so with ease.

International money transfer

One of the main advantages of cryptocurrency is the lack of borders, and instant and automatic transfer of money.

You can for example use cryptocurrency to settle balances with friends, family and co-workers. It can be as simple as scanning a QR code on your phone and confirming the amount to transfer. Really easy.

Plus no extortionate international transfer fees. If you send it to your next door neighbour, or your cousin on the other side of the world, the cost is exactly the same. If you use a no fee cryptocurrency like nano, there is no fee at all, ever!

It will improve

The fact is that this situation will only improve. The methods by which you can utilise cryptocurrencies will get simpler to use, and more common. There is already a lot of research being done, and deals being made with large companies, and even banks. This should give you an idea of how important and revolutionary the technology is.

Even if you plan to wait a little to get fully involved, I think it makes a lot of sense to get an understanding now. This will allow you to be ready and prepared.

What do all the new words mean?

Along with new technology usually comes new terminology. I will give you some quick definitions below before we dig in.

Blockchain

In simple terms the (or a) blockchain is a database where all the transactions are stored (exactly like a traditional ledger).

This database is stored at locations called nodes. A node is just a computer that contains a complete version of the database. There are thousands of nodes around the world, and they all contain exactly the same copy of the database. This means that the system does not exist in one place but is "distributed" around the world. This makes it impossible for governments or other authorities to shut down the system, as it doesn't exist in only one place. If they shut down one node, there are thousands of others.

It is this decentralisation, autonomy and automatic error checking ability of cryptocurrencies that make them so unique.

These nodes talk to each other using a cryptographic algorithm, which ensures that they all agree and hold the same data, essentially forming a network. Furthermore, the cryptographic algorithm ensures that no false transactions can be recorded, and no values can be altered after they are accepted by the network.

If someone tries to change values in the blockchain, or add erroneous data to the blockchain, there are cryptographic methods in place (which I won't go into here) that can spot and reject these transactions. It is this decentralisation, autonomy and automatic error checking ability of cryptocurrencies that make them so unique.

Compare this to a bank. You have a system that is run by humans (notoriously useless error prone creatures) in a central location. This is usually overseen by governments / dictators, which depending on the country could be good or bad. Everybody trusts their bank though don't they?!

I should point out that there are some cryptocurrencies that use something different to a blockchain. However, for all intents and purposes they do the same thing, record transactions. I will therefore stick with blockchain in the article to keep things simple.

Private Key

A private key is literally a string of random letters and numbers like this:

5Kb8kLf9zgWQnogidDA76MzPL6TsZZY36hWXMssSzNydYXYB9KF

It is essentially your password to your money stored on the blockchain. Your private key should be known by you and you alone. Literally no-one else. Treat it exactly as you would your banking login credentials

Public Key

A public key looks a lot like a private key, but tends to be a bit shorter. The key below is my personal Bitcoin public key:

37yi7FUNwv5PQvRU1Q2JEmPYeH8XVyPDTL

The fact that I have published my own public key should tell you that the public key is safe to give to people. In fact you will probably want to give it to people. The public key is effectively the equivalent of your bank account number and sort code (or IBAN if we go international). It gives people all the information they need to send you money.

Wallet

A wallet is where you store your cryptocurrencies. You may hear the terms "software" and "hardware" wallets. They do the same job, but one is stored on a physical device you carry around with you, and the other is stored as a computer file. Typically hardware wallets are considered to be a more secure way to store your wallet information, and hence your cryptocurrencies.

Wallet Seed or Seed Words

The wallet seed is as important to keep safe as the private key. It is created when you make a wallet, and is effectively a more recognisable form of the private key.

Looking at how long and random the private key above is, you can see that it would be very easy to make a mistake copying it or writing it down. It is also not exactly easy to memorise. The wallet seed is a list of (typically 25-ish) words that are commonly used in English (or other languages in some cases). The order of the list of these words is also fixed.

Although a list of 25 commonly used words doesn't sound very secure or random, it is actually statistically very, very, very unlikely that one person will get the same list twice. Especially when you consider the unique order required.

...so should you ever need to recreate your wallet, the ONLY thing you need are those 25-ish words, and you will have access to the same account again! You could also use the private key, they are one and the same just in different formats.

Altcoin

An altcoin is essentially anything that isn't either Bitcoin or Ethereum (i.e. the two biggest cryptocurrencies). It stands for alternative coin.

Mining

Mining is not something you should really worry about too much. It does not happen for all cryptocurrencies, but in some, such as bitcoin, it is used to ensure that the currency stays secure and free from malicious attacks. You could mine cryptocurrency yourself and earn "free" coins, but it is not generally worth the effort as you will earn very, very little unless you have significant computing power at your disposal.

How do I get some cryptocurrencies?

Currently one of the problems with the general adoption of cryptocurrencies is that it is not necessarily obvious how to get hold of them. Even if people know, they are wary as they think they might lose their money.

With the above in mind, I will explain how to get hold of some cryptocurrencies, and make some suggestions on which websites to use to achieve this.

Currency Exchange

The best way to get some cryptocurrency, and start using them, is to exchange your normal fiat currency in your bank account (dollars, euros, pounds etc.) using an exchange website. It is like signing up for any online account. Although you may have to provide some form of ID before they let you transfer money into the account.

As there are many exchanges out there, and there have been various scandals with cryptocurrency exchanges over the years, I will make some recommendations, all of which I currently use or have used in the past.

Kraken

Kraken is my number 1 pick for an exchange that you can convert fiat into cryptourrency. It has a reasonable choice of cryptocurrencies. The only real downside to this exchange is that you will only have access via their website. I believe there is an iOS app available, but apparently it is not very good, and there is no android app available at all.

Coinbase

Coinbase is one of the biggest exchanges that will convert fiat into cryptocurrency. It is very easy to use, and has both iOS and android apps available. However, they have a limited amount of different cryptocurrencies available.

Once you have your account open on one of the two above, buying cryptocurrency is literally clicking a button. You won't struggle, believe me.

Binance

If you start with one of the two exchanges above and find that you want access to more cryptocurrencies, then you can consider using Binance. Binance has a lot of different cryptocurrencies available, is very easy to use, has iOS and android apps, and an impressive array of security features to help you secure your account. The only downside is that you cannot send fiat currency to the exchange. It only deals with cryptocurrencies. This ultimately means you will need to send fiat to kraken or coinbase first. Then change the fiat into cryptocurrencies, and then transfer them to Binance.

The above may sound long winded and a bit of a pain, but it is actually a fairly simple process once you have tried it. I would suggest you transfer a small amount of money to the exchange of your choice and have a play around till you get a feel for it.

Cash

The other option is you skip the exchange, and you buy some cryptocurrency from someone else with cash.

I would not recommend that you go down this road unless you know what you are doing already (i.e. you know how to confirm they have actually sent you the cryptocurrency or not) and absolutely trust the other person involved, or you have no other choice.

Keep things simple and go with the exchange method above.

If you insist on going down this route, you will first need to create a wallet (which I talk about in a later section). Then you simply get them to transfer the funds to your wallet public key address, you confirm the funds are in your wallet, then you give them the cash. Simple. But you MUST confirm the funds arrived in YOUR wallet before giving (or transferring) them any cash.

There are too many coins to choose from which do I pick!?

This is a good question. There are literally thousands! If you don't believe me take a look at CoinMarketCap, which lists them all.

As I stated at the beginning of my article, I am not interested in which one may go up the most so you can make money. What I am interested in is which coin has the best features for using it as money on a day to day basis.

Unfortunately, due to the volatility (which I will touch on later) that we still see on a day to day basis with all cryptos, there are none that I would recommend you plough all your money into. However, there are some that cover a lot of other bases.

Bitcoin

This is the obvious first choice. It is the original cryptocurrency, and as such, is the easiest to get hold of. It also has the most options in terms of where you can use it to buy things. I don't personally think it represents the future of cryptocurrencies, but for now it is a safe bet.

Nano

This coin is a little less known, but it does have some features that make it a potential future winner in terms of use as a real currency:

Transactions are completely free (unlike Bitcoin and most (if not all) other cryptocurrency coins, which all have small transaction fees)
It has implemented wallets on desktop, android and iOS making it easy to use (see the next section for more on wallets)
Transactions are, near as makes no difference, instant
Hardware wallets, such as the Ledger Nano S, can be used to increase security

Monero

This coin is the original privacy coin. Transactions are for all intents and purposes completely anonymous, making it great for people who need some privacy (think overbearing governments, dictatorships, law enforcement etc.).

As a downside I would say it's not the easiest to use. However, that is changing quickly with the introduction of mobile and lite wallets, and also the integration of hardware wallets.

The three I have mentioned above represent what I consider to be projects that serve a purpose. They also seem to have good teams behind their ongoing development (or in the case of Bitcoin are just the biggest at the moment). There are other big players like Ethereum, Litecoin, Ripple etc. With a little research you may find a coin that fits your needs better, but for now the above is what I think represent some projects that have the potential to achieve some real world success.

Wallets

One of the fundamental things to consider with cryptocurrencies is security. This is where wallets come in.

Wallets are basically a place where you store your cyptocurrencies, exactly the same as your bank account. Now you may say "I already have cryptocurrencies stored on the exchange (Kraken, Coinbase, Binance etc.)". You would of course be correct, it is stored on the exchange, but they shouldn't stay there...

Problems with keeping money on an exchange

Firstly, you don't own the private key that stores the cryptocurrencies, the exchange does. What this basically means is that if the exchange disappears over night or decides to lock you out for whatever reason, you have no way to get your money! This has actually happened before, so don't think it is impossible. I'm not saying it is very likely, but just bear in mind that it can happen, and it's too late when it does.

The second problem is that if you want to try and spend some of your cryptocurrencies, in a shop for example, you can't do that efficiently from the exchange, as exchanges are not designed with this feature in mind.

NEVER keep more cryptocurrency on an exchange than you need to, keep them in a wallet that YOU control.

So what is the solution...Wallets!

Wallets allow you, and you alone, to have access and control over your money. You will own the private key, and the private key is the only bit of information that can access the money in the account. You will also have a public key that allows people to send money to you.

Furthermore depending on the specific cryptocurrency, you may have access to specialised wallet features that enhance usability or security.

In essence always follow this rule:

NEVER keep more cryptocurrency on an exchange than you need to, keep them in a wallet that YOU control.

What exactly is a wallet then?

The title wallet is maybe a little bit misleading, as the wallet that you create (or own) actually contains no cryptocurrency at all. It is essentially the information that allows you to access the money that you "own" on the blockchain (a blockchain is just a big, publicly available, secure and immutable database as we discussed in a previous section).

The best analogy is your bank account. You have a bank account number so that people can send you money (equivalent of your public key of your wallet), banking login credentials and debit card so you can send people money, and manage your funds (equivalent of your private key of your wallet). However, the money you have in your account is just an entry in the database of the bank (blockchain of that particular cryptocurrency).

Your private key should be known by you and you alone.

A wallet can then be stated as basically the information needed to access your account (private key), send money (private key) and receive money (public key). There is no actual cryptocurrency in your wallet, it is always stored remotely in the blockchain.

Therefore, a wallet consists literally of a private key and a public key. Which are two different strings of random numbers and letters. Which when you think about it is very, very simple.

What is important though is HOW this information is both created and stored. This is where the types of wallet come in.

What wallet "types" are there?

There are basically three different types of wallet. Hardware, software and paper. They all do exactly the same job, all it comes down to is the balance between convenience and security.

Software Wallets

Software wallets are the most common and convenient way to store cryptocurrencies, but they are also the least secure. They take the form of android or iPhone applications or desktop applications. They can also be accessed through a website.

For example if you want to create a nano wallet you can download the nano wallet app from the playstore or apple appstore. The app will then take you through the process of creating a new wallet step by step.

At some point it will show you a set of seed words that you should copy down and keep safe. This will allow you to regenerate the wallet should you lose your phone or device where the wallet is stored.

Security

The thing that makes software wallets less secure than other methods is that the private key is stored in the software. Therefore, if someone was able to compromise the software, then they might get your private key. However, in practice this is unlikely.

The other possibility is that someone has access to your device when you create the wallet, and copies your seed words. This is more likely than a wallet hack, and you should make sure that you only create wallets on devices you know and trust.

Hardware Wallets

Hardware wallets are the most secure, and in a lot of ways the easiest to use. If I had to recommend a way of storing your cryptocurrencies it would, without a second thought, be on a hardware wallet. Furthermore, if I had to recommend a specific hardware wallet it would be the Ledger Nano S.

A hardware wallet is basically a small usb device that you can plug into your computer or smartphone that lets you send, receive and manage your cryptocurrencies. The thing that makes the device so secure is that the private key is kept inside the device, but it never leaves the device. Not even you know what the private key is, and you don't need to know, as long as you have the device.

What if I lose it!

You may be thinking "but what if I lose the device!". Well this is where the seed words come in. When you set up the device you are given a list of approximately 25 small words to write down. This is only done once, and the words uniquely identify how your device was initialised during setup. Therefore if you ever lose your device, you can enter the same 25 words into a brand new hardware wallet, and it will be initialised identically to the one you lost.

You can then continue as normal as if you never lost the device. The private key generated inside the device will be exactly the same.

As you would expect the 25 words are therefore very important and should be kept very securely in multiple locations as a backup!

How do I use it?

In simple terms you plug the device into the computer and confirm a transaction by clicking a button.

What actually happens is that the device "signs" the transaction with your private key inside the device. It then sends the signed request to transfer funds to the blockchain. The transfer of your money from your funds that exist on the blockchain, to someone else's account on the blockchain, will only succeed if it was your private key that signed the request.

As the signing occurs inside the device, your private key is never exposed, and so cannot be stolen! (Please note that the transfer of funds occurs on the blockchain, there is no actual money inside the device).

Another advantage is that you can access more than one cryptocurrency on a hardware wallet, so you only need one device to manage all your cryptocurrencies.

Please note that although the hardware wallets can deal with a lot of different cryptocurrencies, they cannot deal with them all, so be sure to check your preferred coin is supported before buying the device.

Paper wallet

A paper wallet is probably the most secure way to store funds for long periods of time. Unless you are extremely paranoid I would say this method is not necessary. Either the hardware and / or software wallet will cover most use cases.

To make a paper wallet, run the software needed to create the wallet on a computer that is not connected to the internet (the desktop software for each coin is typically available from the official website of that particular cryptocurrency). Then write on a piece of paper the private and public keys. You then format the drive of the computer used to generate the keys, so the only copy of the private key in existence is on the paper.

With this method there is no way for anyone to get access to your money, unless they find and copy the piece of paper. You could keep it in a safety deposit box for example. It is also recommended to make multiple copies in case one is lost or destroyed.

This method is only good if you want to store money for a long time. As to send money you need to use your private key. The only way to use your private key is on a computer connected to the internet, so you can access the blockchain to confirm the transaction. As you *may* have exposed your private key when you used the private key online, you will likely then need to transfer the remaining funds in the original wallet to a newly generated paper wallet to ensure security is to the same level.

In reality this method only gives you a little bit more security than a hardware wallet, so in my opinion is not worth the effort.

Recommendation

In terms of which wallets to use I would therefore recommend the following:

Software wallets should be used for a limited amount of funds used on a day to day basis for convenience (think current account)

Hardware wallets should be used to store the bulk of your cryptocurrencies (think savings account). Then use the hardware wallet to top up the software wallet when required.

Paper wallets are for people who wear tinfoil hats, or have vast, vast amounts of wealth.

Volatility

Cryptocurrencies are very volatile at the moment. I would expect this to be the case for a while.

This is mainly down to the fact that the general populous don't use them. Plus, the majority of the people that do use them seem to only use them as a commodity to make money, rather than for their core technology.

As the technology is taken into the mainstream this will change. It will also become clearer which of the thousands of coins that are currently available are going to come out on top, and be used by the general populous.

You could draw a parallel with MySpace vs Facebook: the initial idea was floated by Bitcoin (MySpace), but that does not mean it will be the winner in the end as [insert future top crypto here] (Facebook) will come out on top. Better alternatives may arise, and in the interim there will be many contenders for the crown.

Conclusion

Personally, I am convinced that cryptocurrencies will be around permanently in one form or another. What form that eventually takes is anyone's guess, but rest assured they are not going to disappear soon.

I am excited to see where it could take us in a few years time when the technology is a bit more mature, and some of the not so great ideas have fizzled out.

I therefore think it is important that people generally have some kind of grasp of how the technology works. As with so many things in life, the best way to learn is to get your hands dirty! With that in mind I would suggest not putting all your savings into cryptocurrencies just yet, but certainly getting a feel for them is well worth your time.

If anything in the article is unclear, or you want some more clarification or guidance on something I've covered, please don't hesitate to write a comment below and I'll do my best to get back to you ASAP.

Also...on the off chance that my article manages to get you up and running with cryptocurrencies, you could always experiment by sending a small amount my way! Check out my donation cryptocurrency addresses at the link below (and my android app Wanderfile as well if you like):

Cryptocurrency Donations

Happy spending!

How to Make Broadcom Bluetooth Work in Linux (Fedora, Ubuntu)

Sun, 30 Sep 2018 01:00:00 GMT

In my experience Broadcom bluetooth works out of the box on Linux (unlike Broadcom wifi). However, after some system updates it has ceased to function, which is quite an annoyance when your mouse works via bluetooth!

There is however a way to fix this issue and it basically involves resetting the correct driver.

Notes and Pre-requisites

First I want to mention a few of points:

to complete this tutorial you will need an internet connection
this process has been tested on Fedora 26, 27 and 28, and I know it to be working for those systems. It should also be relevant on Ubuntu and its derivatives
if (like me) you have a bluetooth mouse, get a usb mouse temporarily!

Let's get started

Open a terminal and type the following command:

dmesg | egrep -i 'blue|firm'

The output will contain various information, but there will be a particular line that will detail the bluetooth driver that is missing. It will look something like this:

BCM20702A1-0b05-17cf

In your case the letters and numbers might be a little different but the format will be the same. Note this down because this is the name of the driver that you need.

Time to get the drivers

There is a github repository that conveniently has all the Broadcom bluetooth firmware files available:

Direct link to download

Download the repository at the above link and then unzip the file.

You will now have a directory with bluetooth firmware files in it (within "broadcom-bt-firmware/brcm" of the unzipped file). Go through the files and find the file that matches the output you got above. Be careful because the files have very similar names.

Now copy the file to the following directory, and also change the filename to "BCM.hcd":

/lib/firmware/brcm

...or assuming you have the files in your Downloads folder run the following command (replace "filename" with the actual filename you have chosen that matches your Broadcom firmware):

sudo cp ~/Downloads/broadcom-bt-firmware-master/brcm/"filename".hcd /lib/firmware/brcm/

Now reboot...

Open a command line and run:

dmesg | egrep -i 'blue|firm'

This time there should be no error, but if there is then run the following:

sudo modprobe -r btusb && sudo modprobe btusb

Now your bluetooth should be up and running!

Credits

Information in this article is predominantly taken from this source.

What Are Hardware Security Keys Like Yubikey for, How Do They Work, and Do I Need One?

Fri, 02 Nov 2018 00:00:00 GMT

Hardware security keys are about to become the next big thing in online personal security. They provide a simpler, and more secure, way to protect your important online accounts. . .and no more passwords (at least eventually)!

As with all things new it is a bit mysterious and scary sounding to begin with, but for the majority of people it really is very simple and easy to implement. Let's get started. . .

What is a Security Key?

A hardware security key is essentially a physical device (usually a usb stick) that contains cryptographic keys that allow you to log into your accounts by just plugging it into your computer or device. There are also NFC versions that work with smartphones.

Sounds simple, but this method is a LOT more secure than password based access to your accounts.

Do I need a hardware security key?

Of course not, but you really, really should consider getting one. It will give you peace of mind that your accounts and data are REALLY secure.

Google has 85,000+ employees and NONE of their work accounts were maliciously taken over, or used by unauthorised persons, after the introduction of hardware security keys.

Just to give you a solid example of how effective this technology is at combating account hacks and stolen data, consider this:

It has been reported that with the compulsory introduction of hardware security keys at Google the number of employee accounts that were compromised was. . .wait for it. . .ZERO. Google has 85,000+ employees and NONE of their work accounts were maliciously taken over, or used by unauthorised persons, after the introduction of hardware security keys. For more details please refer to this article over at krebsonsecurity.com.

Consider also that these companies all use hardware security keys currently to secure their work systems:

Companies that use U2F

How do I use it?

Simple application

You can actually use hardware security keys for many applications, but the one most people will want is to log into an account. For example your gmail account or Facebook account.

FIDO U2F

At the moment the protocol used is called FIDO U2F, which is also referred to as second factor authentication.

Let's say you have a gmail account that you currently log into with a password. That will not change, you will still use your password as normal, but after you enter your password you will also be requested to plug in your hardware security key. Once that is done you will be logged in. Simple!

The setup process is very straight forward. You log into your account and go to the privacy and security section of your account in settings. Go to the section that is called Second Factor Authentication, and you will be guided through adding your key. It will literally ask you to plug it in at the appropriate moment. Very simple. This video shows how simple the process is:

FIDO2

There is also a new protocol that has just been released called FIDO2. At the time of writing I don't know of any company or website that actually uses FIDO2 for logins, but it is just a matter of time. There are hardware security keys available that work with FIDO2 now, so you can be prepared (for example the Yubico Secrity Key 2 and the Yubico 5 Series).

The main difference between FIDO U2F and FIDO2 is that FIDO2 does not need a password at all, just the key!

Not only will you not have to remember passwords anymore, but it will be more secure and faster as well. You really can't lose. . .but you will have to wait a little until it becomes mainstream.

More complicated applications

If you are very tech savvy and have requirements beyond FIDO U2F and FIDO2, then some hardware security key devices also have the ability to use the following protocols:

OpenPGP
OATH-TOTP
OATH-HOTP
Challenge-Response
Storage of long password strings

This can allow you to achieve many things such as sending files and emails more securely, log into remote servers, log into windows, the list goes on...

Can I use it on all my devices?

Yes you can, but this depends on how you connect the key to your device.

Most keys use a standard USB Type A interface like this:

Therefore you wouldn't be able to use it with a smartphone, or a device that only has a USB Type C port. Although there are a range of different devices available to cover those needs as well, including a NFC interface and USB Type C.

If you want an in depth view of the different types of hardware security key devices available check out my related article here.

What makes it so secure?

The process it uses to confirm the key is yours, and only yours, is what makes it so secure.

In basic terms, it uses something called private-public key cryptography using the method of challenge-response.

What you effectively have is a private key (which is a long string of random letters and numbers) on the hardware security key, and a public key (a different long string of random numbers and letters) on the hardware security key. You "send" the public key to the service (e.g. gmail) when you register the hardware security key.

When you try to log into your account using the hardware security key, the website recognises which device it is (based on the public key, which is unique to your hardware security key) and sends a "challenge" to the device, which is created from a calculation made with your public key. The hardware security key then "solves" the challenge internally and returns the result, which the website checks against the public key. If the public key says the response is correct then you pass, otherwise you fail.

There are a couple of items to note during this process which make it secure:

Your private key is on your hardware security key and NEVER leaves the hardware security key. Your hardware security key only sends a "response" back with the solution to the challenge, never the private key. That means there is (practically) no way for someone to know what your private key actually is. Not even you will know! That means it can't be compromised, ever.
The correct response to the challenge can only be generated by your private key, nothing else.
The challenge is different every time, so even if someone intercepts the response, they can't use it later to get into your account.

Are there any downsides?

There are a few things to consider.

Where you can use it

At the moment you cannot use FIDO U2F everywhere, for all your accounts. It is widely used by a lot of major websites and companies, but not universally. This means you won't be able to make everything very secure just yet. Some of the places I use it at the moment (there are more than this available):

Facebook
Twitter
Dropbox
all google accounts
personal email accounts
github

Browser Support

U2F is only supported by certain browsers at the moment, which limits its use if you don't want to use one of these:

Chrome
Firefox
Opera

Although others are likely in the pipeline.

What if I lose it?

Ah yes! this is a good point. If you setup a hardware security key on an account, then lose it, you will be in a bit of a pickle. However, it is common practise (i.e. it is forced on you by the website you are trying to activate the hardware security key with) to insist on having a backup second factor authentication method setup.

I recommend one of the following:

Setup two hardware security keys

The absolute best solution is to have (at least) two hardware security keys. You can typically setup more than one hardware security key on the same account. This way you can carry one with you, and keep the other in a safe place for emergencies. This will allow you to get into your accounts should you lose the main hardware security key.

Alternative two factor authentication method

If you don't have this option, you should setup a different two factor authentication method which uses a TOTP (time based one time password). This is typically available as an option of two factor authentication wherever a hardware security key is accepted. If you decide to go down this route you will need an app or program that generates the passwords, which are typically 6 digit numbers, and change every 20 seconds or so.

I recommend Authy as it is available on android, ios and desktop. It will also keep things backed-up should you lose your phone / laptop which the app is installed on.

Which hardware security key should I buy?

Brands

There are various hardware security keys available on places like amazon (I can't vouch for their credability), but the leader in the market is Yubico, which produces the Yubikey. The only other "big" player at the moment that I am aware of is Feitian if you really want an alternative.

However, being one of the initial members of the FIDO Alliance, who set the standards for U2F, I would say Yubico are a solid bet when it comes to hardware security keys.

Their main products are also high quality in terms of build, with the standard hardware keys featuring waterproof and crush-proof designs.

If you want a more in depth look at the Yubico hardware security keys, and also recommendations on which one you should buy to suit your needs, then check out our Yubikey article.

Conclusion

Hardware security keys for securing your accounts are the future. They effectively eliminate account takeovers, and will eventually make keeping track of your numerous online accounts manageable, without the hassle of remembering passwords.

I think FIDO U2F and FIDO2 protocols have achieved what a lot of previous attempts have failed to, and that is to make the system simple for the average user.

Currently FIDO U2F requires a password AND the physical key, and I expect that is just too much hassle for the masses, but FIDO2 represents a noticeable (and welcome) change to the process by eliminating passwords completely. I think it is the removal of passwords that will eventually see this technology accepted by everyone.

Bring on FIDO2!

VPN (Virtual Private Network) - What Is It and Do I Need One?

Mon, 05 Nov 2018 00:00:00 GMT

A VPN (Virtual Private Network) is basically an encrypted/secure connection to a remote server. This allows you to access the internet anonymously. The question is, if I am a normal law abiding citizen do I need one? The simple answer is yes, and I will explain why in this article.

Why does an ordinary Joe need a VPN?

Let's say you want to access a website. To do that you go to your computer and type in the internet address of the website. When you press the enter button it sends a message to the website to tell it to send you the web page you want. You receive the data, and the web page loads on your computer.

The failing points

Now, for that to happen you basically sent some data down a wire or through the air (wifi) to your router/modem. It then passes the request on to your ISP (internet service provider), who then pass on the request to the internet site. Then it comes back through the same route. Already we have various points at which the information you sent is accessible:

wireless routers can be hacked relatively easily (if you don't believe me take a look at this article)
your ISP can not only see what you are doing, but is probably logging what you are doing too. If a government or the police want to investigate you, they will be able to request this information from the ISP, and they will get it

"Security is actually not so much a question of 'can it be hacked', but 'how long will it take' to hack"

davidgo (SuperUser Contributor)

...but VPN's are just for dodgy people. Right?

At this point you may be thinking "so what? I don't care if they know what websites I visit anyway. VPNs are just for people who want to hide their tracks when doing something dodgy online."

If you are not doing anything dodgy online that is great, but consider this. If someone has access to your router they can:

Hijack your connection and send you to malicious websites that contain viruses
They can log all of your information
They can steal your login information, bank details etc.

Let's leave the relative security of the home setup for a moment, and move to mobile. How many times have you logged into a public wifi network on your phone? Do you trust all of those freely available networks? Do you know that they haven't been compromised? Do you know who operates them?

The answer is probably "no, I don't know", and I suspect generally you didn't even think about it.

You are more valuable than you think

My point is that in today's world most people are connected to the internet most of the time, and your information is valuable, even if you think it is not. There are criminal organisations making millions from credit card details and identity theft. Then if you are unfortunate enough to live in a country with an oppressive government, you have that to worry about too.

It is wise to reduce the potential points of access to your information if at all possible.

So how can a VPN help?

Encryption

VPNs provide an encrypted connection between your computer and a remote server (in this case your VPN provider). It is often described as a "secure tunnel" between you and the remote VPN server.

If you are not sure what encryption is, it basically takes the data you send/receive from your computer, and changes it into something that makes no sense should someone try to intercept it. It only makes sense again once it is decrypted by you or the VPN server you connect to.

This means that if someone hacks your router with the intention of monitoring your online activity, they will only receive nonsense. If your ISP (or the police/government) want to see what you are up to, all they can see is that you are connected to the VPN server. They won't see any web pages you visit, or anything else for that matter.

Where is the encryption?

Once you have sent the website request to the VPN provider, which is encrypted. The VPN server then sends a request to whichever website or server you want to access. The response from the server / website is then sent back to you through the VPN. Now I should point out that although the connection from the VPN to the website *could* be encrypted, it also may not be. The VPN has no control over the security of the connection to the website as it is determined by the website not the VPN.

If you are trying to hide your tracks this may sound a bit pointless, but remember, they can only see the IP address of the VPN server, not your personal IP address. So although they see what you are doing, they don't know who is doing it.

Shared IP Addresses

One final trick VPN providers have is that they don't assign a unique IP address to everybody connected to the VPN. You will effectively share an external facing IP address with a large number of other people. This is important, as it makes it much more difficult to figure out who is doing what.

As an example, I may be able to figure out that you used a particular VPN IP address. Perhaps you visit my website and I log your VPN IP address when you log into your account. This means that I definitely know that it is you as I can match your account login to the IP address. However, due to the shared IP address used by the VPN I cannot assume that all traffic from that IP address is you. It could be any of the hundreds or thousands of other people utilising that IP address.

Are there any other benefits?

Apart from the obvious fact that your data is encrypted, you may find some of these features useful:

Typically VPN providers will have many servers, and usually they are in different countries. This allows you to appear, to the website you are visiting, as if you are from a country you are not physically in. Why would you want to do this? Well people typically do this to access region restricted content such as tv and movie subscription services. (I want to point out that Netflix has recently taken a hard stance against this practice, and at the moment they block VPN access to their content.)
Bypass network restrictions. For example if your work, school or even government network blocks particular websites, you can connect to a VPN, and access will be possible
It can in some circumstances speed up your connection. If your ISP restricts international bandwidth, or throttles certain services you can then bypass this restriction by connecting to a VPN in your home country

Are there any downsides?

In theory you may lose a little connection speed, as you need to route all your internet traffic through a remote server. However, in reality you will hardly notice at all assuming you pick a good VPN provider (I recommend Private Internet Access as a VPN. I go into more detail as to why in the sections below).

Is there anything else I should be aware of?

Absolutely. There is one last thing that is VERY impotant:

Make sure that the VPN provider does not keep logs at all. This means any data that could identify you directly, or when you used the service, including logon and logoff times. Literally nothing should be kept. This ensures that if anybody sends a request to the VPN for information regarding a particular person or connection, they are unable to provide anything. Even if the VPN is raided by the police, there would be nothing to see.

Ok I want a VPN but how much does it cost and is it complicated?

It is not complicated at all, as long as you use a decent VPN provider. Typically they have an app (android and iOS) for your phone/tablet, and a Windows/Mac/Linux program for your computer, which installs like any other program. Connection to the VPN is usually one click!

Make sure that the VPN provider does not keep logs at all.

There are free VPN providers, which will work fine. However, typically they either cap the speed of your connection, cap downloads, or limit the servers you have access to. If you can put up with some of these limitations then I recommend Cyberghosts free service.

If you want an easy to use and unlimited access VPN it is better to pay. It doesn't cost much either. The paid VPN provider I recommend is Private Internet Access and it costs (at the time of writing) $6.95 a month or as low as $3.33 a month if you sign up for a year. I can personally attest to the quality of this VPN as I use it myself. I think it is a small price to pay for the security it provides.

Both of the services I recommend above do not keep any logs of your activity, which is very important.

Further research

If you want an more information I would recommend looking at this article on torrentfreak. They basically sent an email to a large number of VPN providers and asked a few poignant questions. The answers they receive back are a great help in understanding who to trust, and who to avoid like the plague.

Fedora 29 Workstation – What’s New in the Latest and Greatest?

Wed, 07 Nov 2018 00:00:00 GMT

At the end of October a new release of Fedora Workstation was made available: Fedora 29.

I just wanted to give a brief overview of what has changed, and some of the things to look out for.

...so what has changed since Fedora 28?

This is a quick list of the updates and additions. I go into more detail on each point further into the article:

Modularity - this is the big change with this release, and potentially an important one, both now and for the future.
GNOME has been updated to 3.30 - Lots of new features and improved performance. Another important change.
Flatpak support is integrated by default. This (potentially) represents the future of Linux inter-distribution package development.
Inclusion of non-free packages (if you enable them). Can be both good or bad depending on your viewpoint.
New wallpaper. I'm always in anticipation of the new wallpaper, and this one has been released with a dynamic version (more on that later).
TLS 1.3.
Hidden Grub.
Various software package updates (Python etc.)

Modularity

This is a big one. Going forward this has a lot of potential.

One of the big advantages of running Fedora is that it tends to be the linux distribution with the latest software packages. It is considered bleeding edge.

Being at the bleeding edge has downsides

Being at the forefront is great, but it does have a downside, especially if you are involved in software coding and design. For example you may have a software stack that utilises Python 3.6. Then you upgrade Fedora, and it now comes with Python 3.7. . .ooops!

So now you have an issue. Your code is broken until you re-write it to be compatible with Python 3.7 (this is of course hypothetical and could be applied to many things, but you get the point).

What if you could have old and new together?

This is where the modularity comes in.

You can have a self contained module that runs Python 3.6, and a different one that contains Python 3.7. This allows you to keep a stable setup in one module, whilst taking advantage of the newer 3.7 version in a separate module.

This could be used to test your upgrade path, or just try out new features, but the point is you have the choice!

It is like a new form of Long Term Support (LTS)

One of the oft mentioned downsides of Fedora is that it has a six month upgrade cycle, which is pretty rapid by any standard.

Therefore, people often ask why an LTS version isn't offered. The simple answer is that it would require way too much work, and therefore money, to implement.

If you don't believe me you can hear it from the horses mouth in this great interview with Fedora project leader Matthew Miller.

If you really think about it, what LTS provides is stability. Usually, that stability is achieved by not upgrading your operating system for a long time (in the order of years). This allows you to keep the base up to date (think security patches), whilst software versions remain stable.

However, if you have the ability to continue to run the same software versions, even after an operating system upgrade, then LTS is not really required.

The modularity that has been introduced in Fedora 29 gives you inherent stability if you require it, whilst also giving you access to the latest software versions. This enhances your ability to upgrade software smoothly.

Furthermore, it also has the potential to work the other way around. . .there is more on that in the video above, but essentially it would allow Red Hat Enterprise Linux (RHEL) versions (which tend to be more LTS oriented) to take advantage of the more up to date software versions available in Fedora.

Redhat Linux (RHEL)

Modularity is an excellent addition all round, for now, and the future.

GNOME 3.30

GNOME is another of the main updates with regard to the release of Fedora 29. Some exciting and useful features are now available.

GNOME 3.30 now included in Fedora 29

I have picked out some specific new features that are of interest below, but for a full list be sure to checkout the release notes.

Performance

A major criticism of GNOME has always been that it is resource hungry. This has been somewhat remedied in this release, with a reduced RAM footprint and various behind the scenes fixes for memory leaks and the like.

Flatpaks

Flatpaks essentially represent (potentially at least) the future of how Linux software will be produced and distributed on the workstation variants of Linux.

Currently if you want to produce software for Linux workstations you would have to consider the different package distribution types (rpm and deb), which I understand is not particularly straight forward. Flatpaks can be used on any Linux distribution. This simplifies the production process for developers.

In addition, the new version of GNOME features automatic updates of Flatpaks, making the workstation users life easier as well.

You can check out available software at Flathub, which is the central repository for Flatpaks.

VeraCrypt Integration

VeraCrypt (previously known as TrueCrypt) allows you to create encrypted containers (could be a file or a whole drive). Including advanced features such as hidden volumes.

VeraCrypt (formally TrueCrypt)

GNOME Disks utility now has the ability to decrypt and mount VeraCrypt volumes natively.

An excellent addition for the security conscious.

Other Features

Apart from the above there is also:

improved screen sharing
settings now features thunderbolt devices
a more streamlined location and search bar in the Files app
improvements to the GNOME note taking app Notes
minimal reader view for webpages
Windows server remote desktop access with RDP
A new podcasts app
More games

Non-Free packages included

There are a limited number of non-free (i.e. proprietary) software packages available for install with Fedora 29. By default they are not installed, but the option is there.

Uh Oh! This is a slippery slope...

One of the reasons that people in the past have chosen to go with fedora is that by default ONLY open source software is available upon install. With the release of Fedora 29 this is no longer the case, and there will be some people thinking that this is the beginning of the end for Fedora and its open source roots.

Add this to the fact that IBM has acquired RHEL, and you have yourself quite the conspiracy story!

It is likely for the better

The fact is that you can still install Fedora, and without any effort whatsoever be guaranteed that you are still open source only.

Even in previous Fedora releases it has always been a breeze to get proprietary packages due to the presence of the RPMFusion repository. It's not like we are opening the floodgates all of a sudden.

. . .so the limited proprietary software that has been made available is likely just to ease the pain of installing software that the vast majority need anyway.

As an example, I understand Nvidia users have a very hard time due to Nvidia drivers being proprietary, which with Fedora 29 should be a thing of the past!

As for IBM acquiring RHEL, only time will tell. . .

New Wallpaper!

This is my favourite part!

The designs of the Fedora wallpapers are something to look forward to. Elegant, simple and imaginative without being tacky or distracting.

Here is the new Fedora 29 wallpaper:

Fedora 29 desktop background

However, this time there is a twist. . .you can also have a dynamic version of the wallpaper!

What this basically means is that as the day passes by, the colours in the wallpaper gradually change to reflect the time of day. It is a simple but beautiful concept. Here are the four wallpapers that it graduates between as the day wears on:

Fedora 29 - Dynamic Wallpapers

You will however have to install and activate the dynamic wallpaper manually, so run this:

sudo dnf install f29-backgrounds-animated

. . .and then go and activate the wallpaper on the desktop. The dynamic wallpaper is indicated by a (very) small clock in the bottom right of the thumbnail image.

If you want access to the picture files then check out the Fedora 29 wallpaper github repository, which contains them all.

I believe this has featured before, but as far as I am aware the last time this happened was back in Fedora 26. Anyway, enjoy!

Grub Behaviour Update

Grub is what you see on boot telling you the Fedora kernel version you are about to boot into.

From Fedora 29 this will no longer show if the only thing you have installed on your system is a single Fedora install. This will speed up the boot process.

General Updates

There have also been the usual upgrades to various software packages to keep you right up to date. Including the following:

TLS 1.3 is now supported
Python 3.7
Golang 1.11
Perl 5.28
Ruby on Rails 5.2
Node.js 10.x
MySQL 8

Conclusions

All in all I think Fedora 29 has been an impressive release, and it sets an excellent tone for the future.

You should also bear in mind that the items covered in this article are primarily concentrated on the Workstation edition.

However, the Fedora development team have been making great progress with what could be the next generation of desktop in the form of project SilverBlue. In addition, IOT support has been enhanced by enabling support for ZRAM support for swap on ARMv7 and aarch64. This opens the door for devices such as the Raspberry Pi for example.

A lot to look forward to in Fedora 30 and beyond!

Android App Directory - Useful and Interesting Android Apps

Thu, 15 Nov 2018 00:00:00 GMT

This is a list of Android apps that I think are the best in class or at least interesting. (I don't own an iPhone, so I can't comment on that. . .)

As I discover new apps I will gradually add to the list below, so be sure to check back regularly.

Last updated: 15th November 2018

My Apps

Wanderfile

Allows you to log where you have been in the world using maps and lists. You can log countries, districts of countries and major cities. You can share a multitude of maps in picture form with your friends / family as well, which are produced on demand. It also provides cost and useful travel information for all countries (Disclaimer: I made this app)

App Link

Website

Email

TypeApp or BlueMail

These two represent the best free email apps on the playstore. I have found nothing better (unless you need specific security measures like PGP or S/MIME).

Now just to clear something up...TypeApp and BlueMail are made by the same people, and operate in the same way. The only difference is the way they look (flat vs material design) and the default settings for some settings.

...so essentially download both and then check which interface layout you prefer. I personally use TypeApp..

App Link

Website

MailDroid

This is my main email app, as it has features that I haven't found in any other app. I would say that most likely if you want a general day to day email app you are better off with TypeApp or Bluemail. However, if you want to take advantage of the extra features (signing and encryption) that become available when you use PGP or S/MIME, then MailDroid is the app for you.

I can understand that encrypting emails may be a bit much for most people as it is just a hassle, but signing emails is actually a great way of confirming to the recipient that the email came from you (and only you!).

You may be surprised to hear that the recipients typically don't have to do anything to confirm signed emails with S/MIME, as it is quite well adopted. Gmail for example will confirm signed S/MIME emails natively without any special settings.

I personally sign emails using S/MIME, as the process is a tad easier and more accessible than PGP, but you could use either with this app. Unfortunately, there is no Yubikey integration yet, but if it appears I may switch to PGP...we will see.

One thing to note is that there are some separate apps (from the same developer) that you may need to download to use features like signing/encryption and app themes. They are both free on the playstore and I will list them below as well.

MailDroid - Free

MailDroid Pro - Paid

Crypto Plugin - Free (for PGP and S/MIME)

Themes Plugin - Free

Website

Terminal Access (SSH/FTP)

Termius

The best SSH/FTP client I have found on android. I can literally manage my server as though I was sat at a computer. The interface is also a breeze to use and navigate.

App Link

Website

TermBot

This is another SSH app like Termius. This app is not as polished as Termius, but it does have one killer feature: you can use an NFC Yubikey (NEO or 5 Series) when paired with the OpenKeyChain app. This means you no longer need your ssh/pgp key on your phone, which greatly increases security.

This is now my go to app for terminal access to my server. It is also completely open source, so you can contribute to it becoming a really excellent app!

App Link

Website/GitHub

Security

SafeInCloud

There are a multitude of password managers on the android playstore, and I have tried a lot. SafeInCloud however is the best I have found.

The interface is clean and simple, and it backs up your passwords (encrypted) to YOUR cloud storage (dropbox, google drive etc.), but only if YOU want it to. Plenty of customisation options and it can be used on Android, iOS, Windows and Mac.

This is also one of those apps that you should really buy the pro version of, well worth it.

SafeInCloud - Free

SafeInCloud Pro - Paid

Website

Authy

A great way to securely store your second factor authentication keys across different platforms. Much better than Google Authenticator.

Although this app is great you should consider hardware security keys, as they are the future. Check out my articles here and here to find out more.

App Link

Website

OpenKeyChain

This app allows the use and storage of PGP keys. This allows you to sign and encrypt/decrypt files and messages. It also integrates fantastically with other apps to allow PGP keys to be used in all manner of ways with android (email, password manager etc.). BUT the best feature of this app is that it allows you to use a Yubikey (with NFC, so currently the NEO or the new Yubikey 5) which greatly increases security as you don't need to store your private keys on the phone!

App Link

Website

File Management

Solid Explorer

Simply the best file manager available for android. Does what it's supposed to and can deal with root. It is not free, but it is worth it.

App Link

Website

Website Directory – Useful and Interesting Websites

Thu, 15 Nov 2018 00:00:00 GMT

These are some of the websites that have impressed me with their content and / or usefulness. This list does not include well know sites such as Reddit, Facebook, Google, Twitter, Instagram etc. because you probably know all those already. Hopefully some of them will be useful to you too.

As I discover new websites I will gradually add to the list below, so be sure to check back regularly.

Last updated: 15th November 2018

ShareIcons: A great resource for finding icons and imagery. You can filter by colours, styles etc. making the searching process less tedious.

Tania Rascia: This is essentially a personal blog by a web developer. It is well written, and very useful on many levels. Tania also has the perspective of changing careers, as she was originally a chef(?!), which means she has a unique perspective. One of the best blogs I have come across.

Pixabay: this website is a directory of free to use images and graphics. All items are released under CC0 Creative Commons, which means they are free for commercial use and require no attribution.

This makes finding images for your website or videos (or anything really) a worry free process. There are other sites that offer free pictures, but a lot of them require attribution, or have other requirements. I have no problem with attribution, but a lot of the time it is a hassle to check the requirements. This makes Pixabay a go to website.

IFTTT: this is kind of an amazing website / app. It allows you to setup links between different accounts. For example, if you post a picture to instagram you can set it to automatically post to twitter, or vica-versa. If you receive an email, you can get your office lights to flash. The combinations are endless!

Guerrilla Mail: this is a website that allows you to use a temporary email address so that you can sign up to websites that require login but you don't want to give real details to. There are of course other websites that do this, but Guerrilla Mail has a few unique features, such as:

scrambled email addresses
choosing your own address and domain
easy to manage password manager should you need one (won't go into details here but this point is interesting...)

Virustotal: checks files or websites against a lot of different antivirus software. This allows you to judge by consensus whether the file or website is actually malicious, or you just got a false positive from your own antivirus software.

HaveIBeenPwned: checks whether your email address or username information has been leaked in one of the many website hacks that have happened over the years. You can also sign up to a mailing list that will tell you if your info is leaked in future hacks.

Down for everyone or just me: this does the simple job of checking whether a webpage is alive and kicking. This allows you to check if the problem of connectivity is at your end or theirs.

GetHuman: this is a great resource that helps you skip the automated phone services large companies tend to use, and get straight through to a human (hence the name of the website). Not only that but it gives information on current and average wait times, best time to call etc. A very useful site.

WhichBook: this website allows you to search for books based on your interests. You set sliders for things like gentle - violent, beautiful - disgusting, sex - no sex, short-long etc. and it will return the best match. A great way to find new reads.

JustWatch: this website helps you find where you can legally watch or buy a television series in your country. For example if you want to watch Family Guy, which streaming services (or online shop services) currently have it in their library, and how many seasons do they have available.

CookingForEngineers: A breath of fresh air from the usual fancy pants cooking websites. This website tells you what to do in simple English, with plenty of pictures. And when I say pictures, I mean pictures of how it will actually look, not the result of food dressing and professional photographers. If you want to make some normal food, this is where you go.

CleanTalk: This site allows you to check if an email address, ip or domain is considered to be used for spam or non-legitimate causes. This is particularly useful to check email subscriptions and membership signups for your website.

MediaBiasFactCheck: with all the "Fake News!" floating around today it makes sense to understand where the media you regularly read or listen to sits on the spectrometer of bias! Plus it might give you an idea of where you sit, which could be an eye opener for some...

BrandColors: Want to know the exact colour palettes different big companies use? This site has you covered.

Attributions

Credit where credit is due: I spend some of my time trawling blogs and YouTube looking for inspiration. Sometimes I come across a great source of information that surprises me. Some of the websites on this list I discovered from lists other people have put together, and if that is the case I will link them here:

ThioJoe produced a youtube video of 11 useful websites. Surprisingly it was a very good list: ThioJoe

Signal – The Only Messaging, Voice Call and Video Call App You Should Use

Sun, 25 Nov 2018 00:00:00 GMT

There are many messaging, voice call and video call apps available that claim to use the latest end-to-end encryption to keep your messages and conversations safe. These apps include Facebook Messenger, WhatsApp, Viber and Telegram.

However, as I will explain in this article not all encrypted messaging apps are made equal, and currently if you really value your privacy (which you should) then Signal is the app you should be using.

Why do I even need to encrypt communications?

This is a fair question.

Are you ok with someone opening your letters?

The best analogy I can think of is that of your post. Would you be ok with someone opening your post in transit? I know I wouldn't want someone opening my letters without permission. In the UK it is actually illegal to open mail that is not addressed to you.

Your online communications should be treated exactly the same. Unfortunately it is not so easy to police digital content sent through wires, as it is physical mail.

Bad people and data collection

If the post analogy is a little weak for you then consider the fact that your data, however bland and unimportant to you, is worth a lot to big companies and criminals.

It is not a secret that large revenues are made from harvesting and processing peoples personal data. Using encryption means they don't have access to your data, so it can't be used by them, or stored in their servers.

your data, however bland and unimportant to you, is worth a lot to big companies and criminals

Which brings us on to the bad people (criminals. . .even some governments). . .they could monitor your communications to gather information about you to clone your identity, or get enough info to access your bank accounts. With controlling governments it might mean persecution for political, religious or sexual preferences.

However, do not assume that you are not affected because you don't do anything bad so you have nothing to hide! There is much more to it than that.

The solution...

One of the best solutions to this problem is end-to-end encryption.

What this basically means is that you write a message, then once you hit the send button, it is encrypted. The only person able to decrypt the message is the intended recipient.

This stops the message being read in transit, by a hacker, your internet service provider, the phone company, the government, the police or anybody else for that matter.

The above can also be applied to voice calls and video calls.

What makes Signal so special?

It is open source

It cannot be understated how important it is that software that involves encryption and communications is opensource.

What opensource basically means is that the computer code that has been written to make the app is available for anybody to review. It is therefore close to impossible for any backdoors or other dodgy code to go unnoticed. Someone would spot it.

It also has the added benefit of allowing experts from around the world to review and suggest improvements to the code, which can only make the overall product stronger.

As they say, many hands make light work...

It does not store metadata

What is metadata?

Metadata: a word used, and heard, very often.

However, I suspect that people don't really know what metadata is exactly, and it is really important to know if you value your privacy.

Edward Snowden leaked an NSA document which stated that metadata collection is one of the agency's "most useful tools"

The easiest way to think about metadata is that it is all the data not contained within the message, phone call or video call. For example it may include:

What device is being used?
Where is the device (i.e. a geographic location) that sent the message or start the call?
How long did the call last for?
What date and time did the message get sent or the call start?
Who was the message / call to?

The data above may not seem more important than the content of the message, but it can build a very detailed picture of a user over time.

Where you are, who you communicate with, who you talk to the most etc.

This can have many implications. For example if the government is monitoring you to see who you are talking to due to your political affiliation, or the police to know your whereabouts at a particular time. . .

Maybe the above are too abstract for someone in a western society who is law abiding? How about these:

they know you were calling / texting a particular helpline, and for how long (alcoholics anonymous etc.)
do you phone sex lines or use sex messaging services? They can monitor how long and how often.
having an affair? calling your co-worker out of work hours regularly? ooops!

I think you get the idea. . .it's not just the content that matters.

Have a read of this Business Insider article for more insight...

Does Signal store any data?

Best heard from the horses mouth, in reference to a request for information on a user by the Eastern District of Virginia:

We’ve designed the Signal service to minimize the data we retain about Signal users, so the only information we can produce in response to a request like this is the date and time a user registered with Signal and the last date of a user’s connectivity to the Signal service.

Futhermore:

Notably, things we don’t have stored include anything about a user’s contacts (such as the contacts themselves, a hash of the contacts, any other derivative contact information), anything about a user’s groups (such as how many groups a user is in, which groups a user is in, the membership lists of a user’s groups), or any records of who a user has been communicating with.

All message contents are end-to-end encrypted, so we don’t have that information either.

The quotes above are taken from a page provided by Signal on their website (appropriately called "Big Brother") that details the types of requests for data they receive and how they deal with them. Take a look if you want a better understanding of how your data *should* be handled.

It uses a tried and tested encryption standard

The encryption used in Signal is based on the Signal Protocol (formerly known as the TextSecure Protocol).

Again it is opensource, and as such has been thoroughly tested.

An official academic paper written in collaboration across three different universities (Oxford University - UK, Royal Holloway University of London, UK and McMaster University - Canada) published in 2017 stated that:

we have found no major flaws in its design, which is very encouraging

I would say the above statement represents a typically cautious academic statement. Essentially this means that as far as they are capable of testing the encryption method, it doesn't have any flaws that would compromise its safe use.

The Signal Protocol is also generally accepted as being trustworthy and secure within the cryptographic community.

It is available on many platforms

You can use it on:

Android
iOS
Windows
Mac
Linux (debian based), but I understand that with Flatpak it can also be used on rpm based Linux systems as well

It can also be your SMS app

Signals can replace the standard SMS application on your phone.

This is a great feature as it means you don't need a separate SMS app and all messages can be consolidated in one place.

However, you also need to be a little bit careful. . .

You need to remember that you can only send encrypted messages to other people that also use signal.

If you send an SMS message to someone not using signal, then it won't be encrypted and will just be a standard text message.

. . .so it pays to encourage other people to use the app too. As it ensures the conversations you have will always be encrypted and secure. Not to forget free, as you won't pay for an SMS message!

What do the others lack?

As I alluded to at the beginning of this article - Not all encrypted messaging apps are made equal.

I thought it only fair to go through some of the big names and point out exactly what they lack. In this section I will mention Facebook Messenger, Whatsapp, Telegram and Viber, which I think just about cover the most used messaging and phone apps currently available.

Encryption is not on by default

Although all the apps generally feature end-to-end encryption, it is not typically enabled by default, you have to opt-in.

This may not seem like a big thing. I mean having a choice is good right?

The thing is that there is no reason not to use encryption. The only feasible argument is speed of communication, as the encryption-decryption takes additional time. However, it is a weak argument, and you will likely not even notice a difference.

The problem is that people who are not tech savvy won't even notice the option is there, and won't use it. Which is not fair on those users, as they could also benefit.

Quite frankly a stance like this stinks. It doesn't instil trust in any way.

Facebook Messenger and Telegram both do not encrypt by default. You must either turn it on or use a specific feature of the app.

Metadata and Stored Data

You will find that one of the biggest downfalls of a lot of "secure" messenger apps is the *other* data that they collect and / or store.

One of the excellent outcomes of the GDPR regulation that was enacted in the EU is that privacy policies are relatively transparent, and detail exactly what data is being held. Therefore if you are checking out a new app the best place to start looking for what they have access to is the Privacy Policy.

Facebook Messenger, Telegram, Viber and Whatsapp (owned by Facebook), all collect some form of metadata that I would consider unnecessary. Remember to check out their privacy policies for specifics.

Unique Encryption Methods

Actually most of the apps use an industry standard end-to-end encryption standard such as Signal. However, there is one exception:

Telegram

Telegram uses it's own encryption method: MTProto

There is of course nothing wrong with Telegram having it's own encryption protocol, but it does not benefit from the wide user participation that an open source protocol would benefit from.

It is also not time tested as many of the widely used encryption protocols are. I don't think anybody would argue with the fact that if a method is exposed to, and tested by, as many people as possible for the greatest length of time feasible, and without failure. Then that method is at the very least solid and secure for the present day.

It has also been the subject of not so glowing research papers from places like MIT, and general user scepticism. Telegram have done their best to refute any claims of weakness or failure in their implementation, but it is fair to say that some people are still sceptical. Including me.

Conclusion

The point of this article was to convince you that Signal represents your best bet when it comes to a free, accessible, secure and trustworthy messaging app when compared to the main competition that you have likely heard of before.

I hope I have managed that, if not let me know why in the comments.

I think it should be an industry standard, and right, that you can call, and message anybody without having to worry about being snooped on.

The final thing I wanted to point out is that potentially there may be other apps out there that are better than Signal. I hope there will be plenty of competition in this field. The more the better, because I think it should be an industry standard, and right, that you can call, and message anybody without having to worry about being snooped on.

If you wish to look at alternatives to Signal then I have listed some other apps that you could take a look at below. I have briefly highlighted any concerns or positives to give you a head start...

For now my recommendation stands: Signal represents (currently) the example of what a secure messaging and call app should be - give it a try...

Potential Alternatives

Threema - Possibly the best alternative to Signal. It is a paid option, but we aren't talking big bucks. The only downside is that it isn't completely open source. However, it has been audited by at least two separate auditors, and passed with flying colours. If you take privacy seriously this is really worth a look.

Wire - some of it is opensource, not all. They keep metadata...

Wickr - as far as I can tell it keeps metadata such as date of last use and device type...but take a look if you like

There are probably more, be sure to ping me a message if you have any other interesting suggestions.

Top 10 Essential Android Security Apps (Not Antivirus)

Wed, 05 Dec 2018 00:00:00 GMT

The internet is not a place where you should let your guard down in this day and age. Even big companies have succumbed to hacks, and had huge data leaks.

So without the expertise of a big company, what can we do to protect our private information and activities from prying eyes?

Well fortunately there are a wide array of apps out there to help you do the best you can to keep your info safe. A lot of these apps will allow you to easily implement best practices in terms of mobile security without too much effort on your part.

. . .so in no particular order:

10: Signal - Messenger, Voice and Video

Signal Website

Signal App

Signal has you covered for almost all messaging, phone and video communications.

It can send messages (including pictures and videos) and handle phone calls (inducing video if you like). You can also do the standard things you would expect from a messaging app such as group messaging.

Encrypted messages, encrypted phone calls and encrypted video calls

There are a multitude of messaging and phone apps that now have end-to-end encryption. However, all messaging apps are not created equal. You may be surprised to find that Facebook Messenger, Whatsapp, Viber, Telegram and plenty of others are not quite as safe and secure as you think.

What sets Signal apart from all the other messaging apps is not so simple to explain, so I wrote a separate article to tackle this very subject.

Be sure to check out the article, as messaging and phone calls probably feature quite highly in your phone usage, so it is one of the apps that has the potential to have the most impact on your privacy.

9: Proton Mail - Email

ProtonMail Website

ProtonMail App

You can think of proton mail as a replacement to Gmail or any other web based email client.

You can now step away from the ads and email harvesting that is known to go on with companies like Yahoo and Gmail.

ProtonMail features the following:

end-to-end encryption
opensource
anonymous signup
no IP logging
no tracking
based in Switzerland
free or paid options available

The encryption means that no one is harvesting your emails for information, as no-one can read them. Not even ProtonMail.

Not only can they now read your emails, but they don't bother to log your ip or track you in anyway. Which is exactly how is should work.

8: MailDroid - Email

MailDroid App - Free

MailDroid App - Paid

Maybe you don't want to trust yet another faceless provider, even if it is as compelling as ProtonMail.

MailDroid is an email client that you control.

You enter the servers and settings and the app fetches and sends your emails for you. What this means is that you could use a whole array of email accounts with this app:

Gmail, Yahoo and others
Work accounts
Your own email accounts associated with your own domain

Now it should be noted that there are a lot of apps that are available on the PlayStore that can do this, so what makes MailDroid so special? Well, MailDroid has the option to use S/MIME and/or PGP signing and encryption.

At this point I will say that if you don't know what either S/MIME or PGP are, or you don't have your own server and domain, then this app is probably irrelevant for you. You should take a serious look at ProtonMail instead as it represents a mail app that doesn't require any setup.

However, I will point out that S/MIME setup is not very complicated (PGP is a whole other ball game!). You can get a free S/MIME certificate from Comodo.

Once setup this gives you the following advantages:

Assuming you have your own server with mail: You own your emails! No remote third party server to worry about.
You can sign your emails with your S/MIME or PGP key. This means people will be sure the email they receive *really* came from you (even Gmail recognises S/MIME signed emails with a green tick, so even if the recipient doesn't use S/MIME themselves, they can still benefit from the extra confirmation.)
You can encrypt emails

I would also add that the app itself is very customisable both in terms of looks and features.

7: SafeInCloud - Password Manager

SafeInCloud Website

SafeInCloud App - Free

SafeInCloud - Paid

This app is essential to me.

There is nothing more important than keeping your passwords random and long. This app helps me achieve that with ease.

There are many password managers out there, but they all typically have some sort of cloud sync feature, which more often than not uses the app providers cloud service. I dislike this as I want complete control over my data. Lets face it, if a big company like Facebook can't secure data properly, then any company is vulnerable.

SafeInCloud gives you the option of cloud sync, and even when it is available it is to a location that you own, not them.

You could for example sync to your Google Drive or Dropbox account, or if you are more paranoid you could use your own cloud implementation such as ownCloud or WebDAV.

. . .and finally:

backups are encrypted (AES 256-bit)
password generator
fingerprint login
android wear
password strength analysis
browser integration
free desktop apps (Windows and MAC)
auto import from other password managers

There are both free and paid versions available. At least from my point of view a password manager is an essential piece of kit, and as such I would recommend shelling out the few quid it costs to get the pro (paid) version.

6: Authy - Second Factor Authentication

Authy Website

Authy App

Second Factor Authentication is just a piece of information (or confirmation) that you provide to log into an account in addition to your password. This could be for example a code received in a text message, or a link that you need to click in an email. However, one of the most convenient, secure and widely used methods is Time Based One-Time Password (TOTP).

TOTP is typically a six digit number that changes every 20 to 30 seconds continuously. This means that even if you enter a TOTP in a website and someone figures out what you entered, it doesn't matter, as next time the number will be different.

Authy stores and shows you TOTP numbers, and changes them as needed every 20 to 30 seconds.

Second factor authentication is something that I would encourage everybody to use if it is made available by the website or service in question. It is not complicated for the user, and it increases the security of your account an enormous amount when compared to just having a password.

There are of course other TOTP apps available including Google Authenticator. However, Authy stands out due to the following features:

it backs up your TOTP codes
it can sync your TOTP codes across multiple devices
it works across many platforms
it is easy to use

If you want to take second factor authentication one step further, and in the future get rid of your passwords too. Then take a look hardware security keys like Yubikey, as they are even more secure and convenient.

5: Private Internet Access (or CyberGhost) - VPN

Private Internet Access Website

Private Internet Access App

CyberGhost Website

CyberGhost App

Private Internet Access (PIA) is a VPN, the reason I have mentioned Cyberghost is that PIA is not free, you pay a subscription. Cyberghost is free.

In my opinion PIA is the best VPN currently available, and you should give it a go. It works on most platforms including Windows, MAC, Android, iOS and even Linux.

A plethora of server locations are available across the world, and it is fast and secure. Furthermore, it is one of the few that keeps no records at all of your usage (very important).

If you want to learn more about why you need a VPN and what to look out for (and avoid) in a VPN provider, then I give more information in this article.

4: DuckDuckGo - Browser

DuckDuckGo Website (this is also a search engine you can use, not just info)

DuckDuckGo App

DuckDuckGo is essentially a search engine just like google, but they don't track you at all.

In the same way that you can use Chrome, which features the google search engine, DuckDuckGo have their own android app too.

Using the app gives you access to well thought out features such as forced https connections where possible, and safety ratings for the sites you visit, which are easily visible in the address bar.

It can also blocks trackers, so even if the website in question wants to track you it can't!

3: Sync - Cloud Storage

Sync Website

Sync App

In terms of privacy, cloud storage is a bit of an issue.

There are lots of options, but storing your personal data on the servers of a company you have no control over is risky at best.

This is another area where I believe encryption of data should be a default, but this isn't even nearly the case with the majority of cloud storage providers.

You therefore have a couple of options:

Provide your own cloud storage on your own server. This is unlikely to be feasible for most people.
Find a provider that respects your privacy and encrypts your data

This is where sync comes in.

Sync’s unique, zero-knowledge storage platform guarantees your privacy by encrypting and decrypting your data client-side (on your computer or device).

Sync is a zero-knowledge encrypted cloud platform. That basically means that all the data you store on sync is encrypted using a key that only you hold, so even if law enforcement request access to your data from Sync, they wouldn't have the means to supply it, as they don't have the key.

Sync also features second factor authentication for extra security at login, is easy to use, and can be used on most platforms (although not Linux).

2: Solid Explorer - File Explorer

Solid Explorer Website

Solid Explorer App (14-day trial with in app purchase)

Simply the best file explorer available for Android.

Easy to use, good looking design, root access (if you need it) and encryption if you need it.

Basically it allows you to easily and quickly encrypt files and folders on your phone. It is really simple to use and can also integrate fingerprint recognition if your device allows it.

I should add that it is not free, but the cost is minimal and you won't find better than this.

1: Cerberus - Anti Theft

Cerberus Website

Cerberus App (7-day trial, purchase in app or on the website)

The final consideration for your device is if it gets stolen!

Then you are in trouble if you have sensitive data on your device. I mean lets face it with access to someones mobile phone you have access to a lot of info. Valuable info. . .

Not to mention the fact that "normal" phones (i.e. not covered in gold or jewels) can cost in excess of GBP 1000. So you might actually want the device back too.

Cerberus is an app that sits on your phone minding it's own business, but if set up correctly and your phone is stolen it can work magic.

You could for example (all remotely):

take a picture of the thief
completely wipe your device
set off alarms
backup your data
locate the device on a map
generally control the device

Again it is not free, but a couple of quid to protect a valuable phone is not a lot in my opinion.

Fedora Directory – Interesting and Useful Software and Tweaks

Mon, 24 Dec 2018 00:00:00 GMT

I tend to use Fedora as my main operating system these days, although I do occasionally use Windows when needs must. If you have never tried a Linux distribution before, then I would recommend Fedora as good starting point. I tried Ubuntu too, but for desktop environments I much prefer Fedora. At some point I might write about why. . .stay tuned.

Below are some of the tools and programs that I tend to install on a new Fedora system. For my own benefit, and hopefully yours too.

As I discover new tweaks or programs I will gradually add to the list below, so be sure to check back regularly.

Last updated: 24th December 2018

Gnome Tweak Tool

Allows you to change various settings in the operating system. For example you can add minimise and maximise buttons on windows (not there by default). You can edit system fonts, the clock, mouse settings and many more.

sudo dnf install gnome-tweak-tool

Animated Wallpaper

There are many things I like about Fedora, but one of the standout features is always the wallpaper. They are elegant and interesting, without being distracting or tacky.

For the most recent release of Fedora 29 they have again provided an impressive wallpaper. However, they have gone one step further, and made a dynamic version which gradually changes colour throughout the day. It's subtle but impressive.

You will however have to activate this otherwise you will be stuck with the static version. Which isn't that bad really, but why not if you can. . .

sudo dnf install f29-backgrounds-animated

I am aware they have done this before, but they don't seem to do it for every release of Fedora. The last I am aware of was Fedora 26.

RPM Fusion Repository

By default fedora comes only with freely available open source software. If you need to get hold of proprietary software then you need to use the Fusion repository.

sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

Removal of PackageKit

This is not really a recommended program, but something I tend to do.

Fedora comes with two package managers: rpm and PackageKit.

rpm is the main Fedora package manager, and my preferred option. PackageKit is a part of GNOME and will automatically update Fedora for you. I have, however, had problems with PackageKit in the past, and I generally prefer the control rpm gives you over package installs.

Technically PackageKit is preferable as it performs installs offline (i.e. when you restart), but the lack of transparency and automated nature of PackageKit does not sit well with me.

I therefore always remove PackageKit to reduce the chance of conflicts and problems.

Disable PackageKit Only

Check PackageKit is running and functional.

sudo systemctl status packagekit

Stop and disable PackageKit.

sudo systemctl stop packagekit
sudo systemctl mask packagekit

Completely Remove PackageKit

sudo dnf remove PackageKit\*

Unzip and Archive Tools

sudo dnf install unzip p7zip

Video Codecs

sudo dnf install gstreamer1-plugins-bad-free gstreamer1-plugins-bad-freeworld gstreamer1-plugins-bad-nonfree gstreamer1-plugins-base gstreamer1-plugins-good gstreamer1-plugins-good-gtk gstreamer1-plugins-ugly gstreamer1-plugins-ugly-free

Video Editing and Streaming

Handbrake is great for converting video formats, kdenlive is a great alternative to Adobe Premiere, and OBS Studio is great for streaming and video recording.

sudo dnf install kdenlive handbrake obs-studio

VirtualBox

For creating virtual environments (note the capitalisation of "V" and "B" in VirtualBox as it matters). Which means I don't need to dual boot windows anymore!

sudo dnf install VirtualBox

Android Studio

Install instructions

You need to increase your inotify watch limit as recommended by google.

cat << EOF | sudo tee /etc/sysctl.d/android-studio.conf
# Increase inotify limit, required by Android Studio : https://confluence.jetbrains.com/display/IDEADEV/Inotify+Watches+Limit
fs.inotify.max\_user\_watches = 524288
EOF

then...

sudo sysctl -p --system

Google Chrome:

Download the RPM package from the chrome website. Then run:

sudo dnf install ~/Downloads/{file\_name}.rpm

Inkscape, GIMP and Scribus

I have grouped these together as they are similar programs used for image manipulation and document creation. The best way to describe them is in terms of the Adobe software equivalents as follows: GIMP (Photoshop), Inkscape (Illustrator) and Scribus (InDesign).

sudo dnf install gimp inkscape scribus

Sublime Text

The best text editor for programming and coding there is.

sudo rpm -v --import https://download.sublimetext.com/sublimehq-rpm-pub.gpg
sudo dnf config-manager --add-repo https://download.sublimetext.com/rpm/stable/x86\_64/sublime-text.repo
sudo dnf install sublime-text

Bittorrent Client

The best so far is qbittorent client.

sudo dnf install qbittorrent

VLC

The best video player around.

sudo dnf install vlc

Insomnia

In my opinion the best REST framework tester there is.

First download the appimage.

As the file that you download is an app image, you can run it without installing it to the system by doubleclicking on the downloaded file. However, first you need to allow the execute permission for the file, so run this in the terminal:

sudo chmod +x ~/Downloads/<AppImage>

Once you double click on the file you will be asked if you want to just run the file, or if you want to also integrate it into the system. It is up to you, either will work fine, it depends what you prefer.

Filezilla

The best FTP program.

sudo dnf install filezilla

Yubikey SSH Remote Access

One of the ways to enhance security access to a remote server is to use a Yubikey security key for login and authentication.

(If you are not sure what a Yubikey be sure to check out my article which explains why you should use one, and what they are).

This is achieved with the use of a PGP key that can be setup and stored on the device.

Within Fedora it can occur that ssh-agent and gpg-agent conflict meaning that the Yubikey is not recognised immediately. Running the following should allow the Yubikey to be recognised within the terminal.

sudo killall gpg-agent
sudo killall ssh-agent
eval $( gpg-agent --daemon --enable-ssh-support )

Kernel Update Errors

Sometime after a Kernel update it can cause errors to occur on boot. Usually you will see some text in red displayed during the boot process.

Although usually this does not appear to affect your ability to use Fedora, the following should get rid of the errors.

sudo dnf reinstall dracut
sudo dnf reinstall kernel-core kernel-modules
sudo dracut -v -f

Attributions

Credit where credit is due: I spend some of my time trawling blogs and YouTube looking for inspiration. Sometimes I come across a great source of information that surprises me. Some of the programs / code on this list I discovered from info other people have put together, and if that is the case I will link them here:

Personal Fedora setup of Robbi Nespu: Robbi Nespu

Django REST Framework (DRF) – Initial Setup and Configuration for Ubuntu

Fri, 18 Jan 2019 00:00:00 GMT

Django Rest Framework (DRF) allows the easy creation of a reliable, flexible and secure RESTful APIs.

The idea of this guide is to take you through the setup of DRF on a live Ubuntu 18.04 LTS server running Apache2.

There are of course some guides detailing setup in a local development environment, but not many that cover what is required on a live server.

Furthermore, a lot of the guides assume knowledge of Python, and how to deal with virtual environments. Python Virtual Environments can be confusing if you have never had to deal with them before, so when I say the beginning I really mean the beginning. . .

By the end of this tutorial you should be able to see the successful install page at your domain, and log into the backend through a browser.

Although this guide doesn't detail the setup of views, models, serializers etc. I will likely include that in a part 2 at some point...

What we will cover

I just wanted to briefly go over what we will go through in this tutorial, so you have an overall picture.

Creation of a virtual environment which is required to install our DRF project in
Installation of all required components and dependencies within the virtual environment
Creation of the required folder structure for the project
Setup of the settings file needed to get the project up and running
Creation of a Postgresql database, and setup of the database to allow the app to function
Creation of a superuser for use in the admin area of DRF
Installation of the static files which allow a "browsable API"
Setup of the API site Apache2 config file

Now obviously the above is specific to Ubuntu 18.04 on an Apache2 server.

However, most of the content will be applicable to other server OS and/or web servers with the appropriate changes to server config files and/or package management commands.

Creation of a Python Virtual Environment

Why do we need a Virtual Environment anyway?

When programming with Python it is considered good practise to always have you projects in a virtual environment.

One of the main reasons for this is that it allows you to keep project dependencies stable for each separate project.

Consider this...

. . .so for example, your server last year may have been running Python 3.5. You started a project back then, and wrote the code using Python 3.5, but not within a virtual environment. Just directly on the server.

When the release of Python 3.6 comes around, you would want to keep your server up to date, so that all the other programs that rely on Python can take advantage of the improvements. Also so any new projects you write are using the latest version.

However, if you update your server, you will likely break your project that was written using Python 3.5!

This means your options are to either update the project, or don't upgrade Python. Neither is ideal. . .

...so Virtual Environments are your friend

Virtual Environments help you avoid this situation.

They are essentially a separate box that you install all the dependencies for your project in, and they stay fixed unless you specifically upgrade them from within the virtual environment.

This means you can keep your server up to date without affecting your project!

Creating our project and Virtual Environment

If you haven't already make sure your server us up to date.

sudo apt update
sudo apt upgrade

Then check the version of Python 3 you have installed

user@server:~$ python3 -V
Python 3.6.7

Create a folder which will hold our project. I am going to work within:

/var/www/

but you can store your project wherever you see fit.

I have called my project "myapi" but you can use whatever name you like. Just replace "myapi" as appropriate:

user@server:~$ cd /var/www
user@server:/var/www$ sudo mkdir myapi 
user@server:/var/www$ cd myapi
user@server:/var/www/myapi$

Create the Virtual Environment

user@server:/var/www/myapi$ virtualenv -p python3 env

If we now look at the contents of the "myapi" folder you will see that a folder called "env" has been created

user@server:/var/www/myapi$ ls
env

Now activate and move into the virtual environment

user@server:/var/www/myapi$ source env/bin/activate

You can see that you are now in the virtual environment due to the bracketed "env" in the terminal at the beginning of the line

(env) user@server:/var/www/myapi$

Whilst within the virtual environment you can use "python" rather than "python3" as a command.

Any dependencies you install will be specific to the virtual environment not the server overall.

Installation of dependencies

Whilst in the virtual environment we will be using Python PIP, which is the Python package manager (just like apt for Ubuntu and dnf for Fedora). This is included by default with Python versions 3.4 and above, so you shouldn't need to install it.

The main two packages we need are "django" and "djangorestframework".

I would also recommend installing the "markdown" package, which enables markdown in the browsable API (more on that later). Plus the "django-filter" package, which is a very useful addition when writing the api, as it allows filtering support.

So run the following:

pip install django
pip install djangorestframework
pip install markdown
pip install django-filter

Setup the project and folder structure

We are now going to setup the project within the virtual environment. This will involve starting a "project" and an "app" within the virtual environment.

This process creates a folder structure, and also produces various Python files that will define your project and app.

Why do we need a project and app?

To give you an idea of the concept of "projects" and "apps" in Django (which Djangorestframework is based on) I'll use a website as an example.

The "project" would be the website as a whole. The "apps" would be sections of the website, for example, blog "app" or news "app" etc.

The apps are separate components that are brought together in a "project", but are effectively self contained and reusable.

In this case there is little point making the distinction as we will only have one app, but it is always good to know why the structure exists.

For more in depth info have a look as the Django Project Structure.

Making the project and app

At this point you should be within the virtual environment at the root of the project folder you created. Like this:

(env) user@server:/var/www/myapi$

Now we will create the "project". Please note that there is a "." at the end of the next command. This is not a mistake. If you omit the "." it will create an additional level of folders that you don't need:

django-admin startproject myapi .

The above command will create a file called "manage.py" and a folder called "myapi". You will therefore have the following folder structure:

(env) user@server:/var/www/myapi$ ls
manage.py env myapi

We then move into the new project folder and start a new app. In this case I will call the app "api"

(env) user@server:/var/www/myapi$ cd myapi
(env) user@server:/var/www/myapi/myapi$ django-admin startapp api

We can now exit the virtual environment. You will notice the (env) disappears.

(env) user@server:/var/www/myapi/myapi$ deactivate
user@server:/var/www/myapi/myapi$

At this point you should have a folder structure that looks something like this (Please note I have not given detail of the "env" folder contents as it is not necessary to know, but it will contain a lot of folders and files):

/var/www/myapi/manage.py
/var/www/myapi/env
/var/www/myapi/myapi/__init__.py
/var/www/myapi/myapi/settings.py
/var/www/myapi/myapi/urls.py
/var/www/myapi/myapi/wsgi.py
/var/www/myapi/myapi/__pycache__
/var/www/myapi/myapi/api/admin.py
/var/www/myapi/myapi/api/apps.py
/var/www/myapi/myapi/api/__init__.py
/var/www/myapi/myapi/api/models.py
/var/www/myapi/myapi/api/tests.py
/var/www/myapi/myapi/api/views.py
/var/www/myapi/myapi/api/migrations/__init__.py

Adjusting the settings.py file

As you can see in the structure above there is a settings.py file located in the project folder. settings.py will have standard settings in it, but I have provided below an updated version to allow us to proceed. Take a little time to compare the two and make sure your file reflects the one below.

Note: it is only necessary to be in the virtual environment when making dependency upgrades or working with Django or Python directly. If you just want to edit the ".py" files this can be done without being in the virtual environment.

You can open the file and edit it with nano text editor (or any text editor you prefer). If you don't have nano installed you can do the following:

user@server:/var/www/myapi/myapi$ sudo apt install nano

. . .and then open the file:

user@server:/var/www/myapi/myapi$ sudo nano settings.py

Do not just copy and paste the file below over your file! Your file will have a "secret key" in it already. You need this, so don't overwrite it.

Remember to make sure you use your own folder paths if they differ from the ones used in this article.

"""
Django settings for myapi project.

Generated by 'django-admin startproject' using Django 2.1.5.

For more information on this file, see
https://docs.djangoproject.com/en/2.1/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/2.1/ref/settings/
"""

import os

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.1/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'removed-secret-key-leave-yours-here'

# SECURITY WARNING: don't run with debug turned on in production!
#leave as True for now, but when you send into production (i.e. other people will use it) then set to false.

DEBUG = True

#where it says 'myapi.thetestspecimen.com' you should replace this with your actual domain name for the api.

ALLOWED_HOSTS = ['localhost','127.0.0.1','myapi.thetestspecimen.com']

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'django_filters',
    'myapi.api.apps.ApiConfig',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'myapi.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'myapi.wsgi.application'

# Database
# https://docs.djangoproject.com/en/2.1/ref/settings/#databases

#We will be using a Postgresql database. Replace the database name, user and password with what you would like to use when we setup the database in the next section of the article

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'dbmyapi',
        'USER': 'myusername',
        'PASSWORD': 'mypassword',
        'HOST': 'localhost',
        'PORT': '',
    }
}

# Password validation
# https://docs.djangoproject.com/en/2.1/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# Internationalization
# https://docs.djangoproject.com/en/2.1/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.1/howto/static-files/

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

#These are some settings that make the setup more secure you don't need these if you don't want them

SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_HSTS_SECONDS = 3600 

#Uncomment the section below if you want to remove the browsable API later on. For now just leave this section hashed out

#REST_FRAMEWORK = {
#    'DEFAULT_RENDERER_CLASSES': (
#        'rest_framework.renderers.JSONRenderer',
#    )
#}
########################################################

Setting up the Postgresql database

There various options of database that can be used for DRF, but I personally recommend Postgres as it is the closest to the real SQL standard, has an extensive feature set, and is open source.

The other options are sqlite, MySQL and Oracle, should you prefer to use one of those.

I'm not going to go into a lot of detail here, but if you want more info on setting up postgres, Digital Ocean have a great tutorial.

First we will install postgres:

sudo apt install postgresql postgresql-contrib

Then we log into the commandline interface for postgres:

sudo -u postgres psql

You should now see a new commandline like this:

postgres=#

We now need to create a user, a database and then assign the user to the database.

Remember to use the same details that you put in your settings.py file we setup in the previous section:

postgres=# CREATE USER myusername WITH PASSWORD 'mypassword';
postgres=# CREATE DATABASE dbmyapi;
postgres=# GRANT ALL PRIVILEGES ON DATABASE dbmyapi to myusername;

Then we can check it has worked by listing the databases.

Note: that there may be other items listed, but we should have at least the below:

postgres=# \l
  List of databases
     Name     |  Owner   | Encoding |   Collate   |    Ctype    |     Access privileges     
--------------+----------+----------+-------------+-------------+---------------------------
 dbmyapi      | postgres | UTF8     | en\_US.UTF-8 | en\_US.UTF-8 | =Tc/postgres             +
              |          |          |             |             | postgres=CTc/postgres    +
              |          |          |             |             | myusername=CTc/postgres

If the databases is listed correctly we can then quit the postgres commandline:

postgres=# \q
user@server:/var/www/myapi/myapi$

We are now in a position to connect the database with django rest framework:

user@server:/var/www/myapi/myapi$ cd /var/www/myapi
user@server:/var/www/myapi$ source env/bin/activate
(env) user@server:/var/www/myapi$ python manage.py migrate

The above command should setup the necessary parameters within the database you have just created.

If there are any errors reported double check that the parameters you setup for the database are correct in the settings.py file.

Create Django Superuser

To allow you to log in and administer the API from a browser you need to create a superuser for the project:

(env) user@server:/var/www/myapi$ python manage.py createsuperuser --email admin@example.com --username admin

You will then be prompted to enter a password for the user as well.

Once the setup has been completed successfully you will be able to access the admin area from a url like this:

https://myapi.thetestspecimen.com/admin/

Obviously the domain will be whatever you choose to setup, which we will get to in the last section.

Download static files

DRF features a Browsable API.

What this essential means is that once you have setup endpoints you will be able to access them from a browser to see how they respond.

Take a look here for more details.

As the browsable API has some static components, we need to download them into our project so that the browsable API renders properly:

(env) user@server:/var/www/myapi$ python manage.py collectstatic

If you then run:

(env) user@server:/var/www/myapi$ ls
manage.py env myapi static

You will see the addition of a static folder. You can then exit the virtual environment:

(env) user@server:/var/www/myapi$ deactivate
user@server:/var/www/myapi$

Setup Apache2 for the API domain

This section you may be able to handle without instruction, but I have included just for completeness.

We are basically going to setup the "sites-available" config file for the api domain. This will tell apache2 where to find the python files it should load when that specific domain name is requested.

In this case the setup file will contain the configuration required for a https connection to the API. This assumes that an SSL certificate has been setup for the domain. If this is not the case, you will have to adjust the setup to work without SSL (which I don't recommend).

You can setup a free SSL certificate using Let's Encrypt, but it can be a little fiddly to get setup and to autorenew. I intend to write an article for this in the future, but for now check out this guide.

In this case I am going to use an example domain of "myapi.thetestspecimen.com"

Set A Records

If you haven't already, go to your server setup and make sure that you have created a set of A records for the domain you intend to use.

If you are not sure how to do this then check your server providers help files, as it varies with provider.

Create config file in sites available

We need to create the config file so apache knows where to look for the project. Obviously use your own domain name not the one below:

user@server:/var/www/myapi$ cd /etc/apache2/sites-available
user@server:/etc/apache2/sites-available$ sudo touch myapi.thetestsepcimen.com.conf
user@server:/etc/apache2/sites-available$ sudo nano myapi.thetestspecimen.com.conf

Within the file paste the following, with the appropriate changes to folder paths and domain names:

#Example Django Rest Framework File

<VirtualHost *:80>
    ServerAdmin admin@thetestspecimen.com
    ServerName myapi.thetestspecimen.com
    
    Alias /.well-known/acme-challenge/ /var/www/myapi/.well-known/acme-challenge/
    <Directory /var/www/myapi/>
    AllowOverride None
    Options None
    ForceType plain/text
    </Directory>
     
    # This is optional, in case you want to redirect people 
    # from http to https automatically.
    RewriteEngine On
    RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

</VirtualHost>

<VirtualHost *:443>
  # Virtual host configuration + information (replicate changes to *:443 below)
  ServerAdmin admin@thetestspecimen.com
  ServerName api.thetestspecimen.com
  DocumentRoot /var/www/myapi

  Alias /static /var/www/myapi/static
  <Directory /var/www/myapi/static>
	Require all granted
  </Directory>
     
  <Directory /var/www/myapi/myapi>
	<Files wsgi.py>
		Require all granted
	</Files>
  </Directory>

  WSGIDaemonProcess myapi  python-home=/var/www/myapi/env python-path=/var/www/myapi
  WSGIProcessGroup myapi
  WSGIScriptAlias / /var/www/myapi/myapi/wsgi.py

  # SSL certificate + engine configuration
  SSLEngine on
  SSLCertificateFile /etc/letsencrypt/live/thetestspecimen.com/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/thetestspecimen.com/privkey.pem

</VirtualHost>

Once you have saved the file we then need to load the config file and restart apache2:

user@server:/etc/apache2/sites-available$ sudo a2ensite myapi.thetestspecimen.com.conf
user@server:/etc/apache2/sites-available$ sudo service apache2 reload

Check the admin site works!

At this point we are done with the basic setup.

We can check if everything we have done worked by visiting the API url in a browser. In my case "https://myapi.thetestspecimen.com". You should see this:

Django rest framework success screen

Furthermore, we can check all the settings worked by visiting the admin site and logging in with the superuser credentials you created earlier.

Going to "https://myapi.thetestspecimen.com/admin" I will see the following (remember to use your own domain when you try it):

Django rest framework admin login screen

The End!

You should now be able to start the journey of setting up your RESTful API.

Django rest framework makes it really easy to achieve an excellent API without boilerplate code. You should be up and running in no time!

Best of luck, and if you have any questions, let me know in the comments or send me an email.

As alluded to at the beginning of the article, I intend to write a part 2 to this tutorial. It will detail a basic setup of the configuration files so you can get some endpoints working. Watch this space. . .

Yubico Yubikey – Which Hardware Security Key Is the Best for Me?

Sat, 19 Jan 2019 00:00:00 GMT

Yubico is the market leader when it comes to hardware security keys utilising FIDO U2F and FIDO2 such as the Yubikey, but which one should you buy to suit YOUR needs?

There are actually a wide array of hardware security keys available from Yubico, and it can be quite confusing to try and pick out what you do and don't need.

I hate purchasing a product and then finding out it is either missing a feature I wanted, or there was something better suited to my needs that I missed...

Update Jan 2019

It turns out that Yubico have recently released an NFC version of their Security Key.

This along with the release of the Yubikey 5 series late last year makes picking up the best key a lot easier. I have therefore updated the article below to reflect this.

Before we begin

If you are looking at this article and you have no idea who Yubico are, or what a hardware security key is, then I recommend you read our previous article which should bring you up to speed.

The Main Yubico Family

Yubico 5 series family plus the Yubico Security Key 2

The picture above shows the main security keys that are currently available form Yubico. The blue device on the left is the Yubico Security Key (of which there are now two versions: with NFC and without NFC), and the four devices on the right are the Yubikey 5 Series keys in different form factors. They all look fairly similar, but in some cases they have very different features, so I will do my best to point you in the right direction.

What's the difference?

All the keys listed above can deal with both FIDO U2F and FIDO2 (if you don't know what either of those are take a look at my previous article), which are the main protocols that everyday users will be interested in at the moment. One of the most important new releases is FIDO2 support, which represents the future of password-less login.

From that point is where the differences start to come in. Some have NFC, some don't. Some have more advanced protocols available, others don't. . .and as you can see from the picture above they come in a variety of form factors.

So let's get stuck in!

Yubikey 5 Series

The Yubikey 5 series is the most advanced key Yubico produce in terms of features. If you want all the possible features you can think of in one security key then these guys have you covered.

Yubico Yubikey 5 series family

In terms of the security capabilities of the series 5 keys, all have exactly the same features. The only differences come with the interfaces used for connectivity. So from left to right:

USB Type A and NFC
USB Type C (no NFC)
USB Type A low profile (no NFC) - designed to be very low profile in a laptop or server
USB type C low profile (no NFC) - designed to be very low profile in a laptop or server

In terms of protocols that the keys can deal with, here it is:

FIDO2
FIDO U2F
OpenPGP
Smart Card (PIV)
Yubico OTP
OATH-TOTP
OATH-HOTP
Challenge-Response
Storage of long password strings

If you don't know what any of the above are apart from FIDO U2F and FIDO2, then it is likely you will not need them. If that is the case you may be more interested in the Yubico Security Key, which we will get to later in the article.

I would encourage you to take a look at the uses for each item on the list above in case the use case interests you. For example OpenPGP can be used to send secure emails, and encrypt and decrypt files.

Which of the four series 5 keys is the best to get?

This depends on your circumstances, but in general you should opt for the USB type A version.

There are various reasons for this. Firstly the build quality of this key is excellent. It is waterproof and crush-proof.

It is extremely thin, and about the same size as a standard house key, which makes it ideal for key-chains.

Thickness comparison of Yubico Security Key 2, Yubikey 5 and a standard house key

Yubico Securtiy Key 2, Yubikey 4 (same form factor as Yubikey 5) and standard house key

...and on top of all that it is the only key in the Yubico 5 Series that has NFC connectivity. This means that you can not only use this device to secure your logins on your desktop computer, but also your android smartphone or iPhone!

Yubico Yubikey 5 NFC for android

Yubico Yubikey 5 NFC for the iPhone

With all that aside, there are obviously reasons you might want to opt for one of the other keys.

For example if you want to have the keys permanently in a laptop or server, then the low profile USB-A and USB-C keys make sense. Also if you predominantly use devices with USB type C ports then the USB type C key makes a lot of sense.

However, as detailed above I think for most people the USB Type A key is just the best all rounder. Rugged, and with the best connectivity options.

Yubico Security Key 2

If your main concern is securing your login into websites such as Facebook, Twitter, Google accounts etc. Then the main protocols of interest are FIDO2 and FIDO U2F.

The Yubico Security key 2 covers those use cases without all the other more complicated protocols. As such it represents great value for most people, at around half the price of the 5 series keys.

Furthermore, as of January 2019 the Security Key can now be purchased with NFC connectivity, so you can use it with your phone or other NFC capable devices.

Yubico FIPS (not pictured)

Yubico has another series of keys that look identical to the 5 series keys. They are called the FIPS series.

As I understand it, Yubico has been seeking certification for FIPS 140 for a while, and it just attained it.

FIPS-140 is basically a set of standards that a hardware security device must meet according to the requirements of US government agencies. The requirements are quite stringent, which proves that the keys on offer by Yubico are indeed top notch.

It was actually the previous series (4 series) keys that were entered for certification, which is why they look exactly the same (please note the 5 series and 4 series keys look literally identical with the exception of the NFC sign on the 5 series USB type A key). What this likely means is that the FIPS keys are based on the old 4 series keys, and so you would miss out on NFC connectivity for example.

The long and short is: you only need the FIPS series keys if you work for a government agency that requires FIPS certification, otherwise it really isn't worth the hassle.

My Current Setup

Just to give you an idea of what I use my keys for. . .

I actually own three Yubikeys. I decided to buy a Security Key 2 (without NFC as there was no option at the time) and a Yubikey 4 USB type A originally (see picture below). I have since purchased a Yubikey 5 Series USB Type A key to take advantage of the NFC capabilities of the new key. For me this is just about perfect.

I have my main 5 Series key that I carry around all the time, which can deal with FIDO U2F and FIDO2 for logins, whilst also holding my PGP keys for server login, and file signing / encryption. I then have the Yubico Security Key 2 as a backup for all the FIDO U2F and FIDO2 logins, and the Series 4 Key contains a backup of my PGP keys.

If I was to do it all again I would likely just get two Yubikey 5 Series keys, but I didn't have that option when I originally purchased them.

Conclusion and Recommendations

For most people I would recommend that you buy at least two Yubico Security Key 2 NFC devices.

If you really need the extra features of the 5 Series Yubikeys, then I would recommend you buy at least two Yubikey 5 Series NFC Type A keys.

If you are not sure why you would need a pair of security keys I would suggest giving my previous article a read, which explains why. The short answer is: it acts as a backup should you lose one!

Overall, my experience so far has been excellent with regard to FIDO U2F. It really is a breeze to use, but I think the guidance and usage with regard to the more advanced features needs a little bit of work to make it easier to implement.

Either way they are worth the effort to learn how to use as the extra security they bring cannot be understated.

Why I Switched to Manjaro As My Daily Linux Operating System

Fri, 14 May 2021 01:00:00 GMT

This article is not to convince Windows users to switch, although for some it may be beneficial. It is to detail why I decided on Manjaro over the long list of other Linux distributions that are available.

I have used other systems such as Ubuntu and Fedora for years, so have a reasonable view of what is going on elsewhere. So if you think another Linux users opinion may be interesting, maybe read on...

What previous Linux distros have you used?

I have been using Linux as my main and only OS for quite a few years now. I started off using Ubuntu, which I suspect a lot of people do. I still believe Ubuntu is probably the best starting point for someone new to Linux. Mainly because of the large community, and hence guidance, available online.

When I got my first itch to try something different I ended up settling on Fedora (which I think was at about release 26 at the time). I then stuck with Fedora right up until switching to Manjaro at the start of this year (2021).

I should also note that over all of that time my server has been running on Ubuntu. The option of 'fully up to date' or Long Term Support (LTS) versions is one of the great attributes that Ubuntu has available.

Have you used any other distributions over that time

The amount of distributions Linux has available could be seen as a curse, and I am sure it is a bit overwhelming for newcomers. However, I have generally found it to be of great use in various situations. These are the distributions I currently use:

Zorin OS

I chose Zorin OS xfce install to replace Chrome OS on my, now dated, Acer C720P chromebook. This was mainly because I was no longer receiving security updates from ChromeOS.

The main problem I had is that it has non-upgradeable RAM, and there is only 2GB available on my particular model. This seriously limits your options.

I basically wanted something that looked nice, and ran smoothly. With the limited power and RAM. Zorin OS xfce met these requirements perfectly.

Raspberry Pi (Raspbian)

I needed to produce something that was cheap but fully functional, and that could integrate with existing windows systems.

Raspian, or more the cheapness of Raspberry Pi, fit the bill here perfectly.

Tails

I always have a tails install on one of my USB drives.

If you need some temporary security while out and about, Tails is perfect.

Others

I have also at various points in time used the following:

Qubes (A great security focused OS)
Pop OS (Probably the best ubuntu based OS out there)
Kali (Designed for penetration testing)

There may be more that I can't remember.

Considering all the distros I have tried, I haven't even really scratched the surface. There are many more!

Why did you consider Manjaro over all the other distros?

Rolling updates

One of the main reasons that I initially decided to look into Manjaro is that I heard it has a 'rolling' update method of releasing updates.

What exactly are 'rolling' updates

What this basically means is that the OS is updated continuously as updates for various programs and packages are released by their maintainers. This differs from other large distros such as Ubuntu or Fedora which have major updates approximately every 6 months.

It is nice to see the shiny new features after a major upgrade, but this is often tarnished by the fiddling about that is required afterwards.

The problem withmajor updates every 6 months is that, more often than not, something will break. These days the operatings system itself will update perfectly. However, every time I have done a major update there is a program or system utility that will need fixing or readjusting to work as it did before the update ran.

It is nice to see the shiny new features after a major upgrade, but this is often tarnished by the fiddling about that is required afterwards.

You don't have any of this pain with Manjaro.

The advantages of rolling updates

You may argue that things can go wrong with rolling updates, and yes they can. I haven't experienced it yet, but I have no doubt my day will come!

The difference is, that with rolling updates you have far less to deal with. Trying to pin point the source of a problem after a major 6 month upgrade affecting hundreds, maybe thousands of packages is a nightmare. The smaller 'upgrade' of rolling releases are much more managable.

It is based on Arch Linux

This more intrigued me than anything else. Arch linux has a reputation, and a good one at that.

Think about the type of person you need to be to run Arch. You need to be really into your system to want to spend the time it takes to basically build your operating system from scratch. And that's what Arch is, ultimate flexibility.

...but this has a side effect. The people involved in the community are likely to be very knowledgable. This is basically proven by the existence of the Arch Wiki and Arch User Repository (AUR).

Even if you don't use Arch Linux, the arch wiki is an insanely good source of information to troubleshoot all sorts of problems!

If you like the idea of Arch, why not just use Arch?

This is where it comes down to personal preference.

I could go the whole hog, but the reality is I don't want to. I have no desire to build my system from scratch, because to be honest when I started looking into Manjaro it seemed to be well thought out, and pretty flexible if you want it to be. I suspect I would struggle to beat it with my own build!

I have no desire to build my system from scratch

There is also an added element of stability. The Manjaro developers test updates released for Arch for stability etc. before they are made available in Manjaro. I think this is a very good idea.

Yes, you sacrafice some cutting-edge-ness, but for me at least, the amount I lose is very small, and irrelevant. I much prefer the added stability.

So you like Manjaro, what made you stick with it?

Suprisingly quite a lot. I really liked using Fedora, and did for many years, but Manjaro fits me like a glove. It really is an excellent distro.

Many install options

People have different tastes when it comes to desktop GUI environments. As main 'official' options you have:

GNOME
KDE Plasma
Xfce

In general I would say that covers most people. Not a bad start!

Don't like those? Well you can also get the following community editions, easily downloadable from the official website:

Budgie
Cinnamon
i3
MATE
Sway

Have an ARM device. No worries! There are loads of options for that too. Even the Rasperry Pi 4!

Further install flexibility if you want it

The previous section should satisfy most people, and the install is a breeze. But what if you didn't like one of the options chosen for you in the install?

I had this problem. I wanted to use Btrfs as the disk filesystem, not ext4 which is standard.

When I originally installed Manjaro at the start of this year there was an 'Architect' version available that allowed a much more fine tuned install process, especially in terms of partitioning, swap, file system etc.

I have noticed that currently this option seems to have disappeared from the official website, but it was possible to boot into the architect installer from any ISO (which is what I did), so hopefully this option remains for the added flexibility that some people will want.

If you don't have any specific additional requirements (and you will know if you have) then the standard installs are sufficient, and really easy to deal with.

Package management

In previous distros I have tended to do all package updating on the command line. This was mainly because I found it easier and more informative. I have also never really been very impressed with the GUI package managers of either Ubuntu or Fedora, in my case in Gnome.

Installing third party packages on either Fedora or Ubuntu (think PPAs if you are familiar with Ubuntu), always ends up being quite messy. It is something that I have put up with, as I didn't see another way, but the truth is I found it quite annoying.

Installing third party packages on either Fedora or Ubuntu (think PPAs if you are familiar with Ubuntu), always ends up being quite messy.

I think the package management, and the tools and methods used in Manjaro, is one of the major things that make it a pleasure to use. If you really think about it, you interact with package managers on a day to day basis. They need to work well.

There are three items that make it work smoothly:

Pacman (the base package manager)
Pamac (the GUI package manager)
Arch User Repository (AUR)

...and unusually, it is the GUI package manager that is the stand-out item in this list. It is the only time I have started to favour the GUI package over the command line.

To understand why it works well, you first need to understand what the AUR is. It is essentially a repository that contains user made packages, a bit like PPAs in Ubuntu. So any packages that you don't have access to from the main Manjaro repo, you can get from the AUR (if it exists). If not, you are free to contribute yourself, and as always there are extensive instructions in the Arch wiki to help you do just that.

As you move forward, you don't end up with a fragmented mess.

The clever part is that Pamac keeps track of both normal Manjaro packages and AUR packages, and they are labelled clearly as such. You can install and uninstall either in exactly the same way - with the click of a button!

Some of the credit for this simple setup needs to be given to the AUR itself, which by nature ensures a consistent way of packaging the files. This means that any dependencies are taken care of as well, so you don't have to go fiddling around in config files, or installing additional items manually. As you move forward, you don't end up with a fragmented mess.

The base installs are aesthetically pleasing

First and foremost the default settings for a desktop environment are really nice in terms of aesthetics. I use GNOME as my desktop, so it is a little difficult to comment of the other spins (KDE and xfce), but I have briefly installed the xfce version, and I would say it is probably the most aesthetically pleasing xfce I have ever seen (with the exception of Zorin OS xfce, maybe).

The above is subjective obviously, and of course you could customise it, as you could on any other GNOME desktop. However, out of the box I prefer the base GNOME setup to any other I have seen.

Desktop layout is highly configurable

More inportant than the above, and another main reason for switching, is the flexibility they have made available out of the box. In this case I am talking about how you like to configure your desktop.

The different layout options available with Manjaro

In the image above you can see the various options.

Manjaro - a slightly tweaked traditional Gnome layout (uses dash-to-dock for example)
Traditional - as close to Windows as you can get
Unity
Modern - think MacOS layout
Gnome - the bog standard Gnome layout
Tiling - an auto tiling desktop

You can switch between any of the above with ease and try out which one you like. This means that if you are not a massive fan of the traditional Gnome layout then you can get things how you like them without installing additional 3rd party packages.

As for me, I went with the Manjaro layout, but moved the dash-to-dock bar down to the bottom rather than the left edge. I never understood the left edge layout, as all your icons end up squashed.

The layout switcher also has a settings tab that gives you quick access to things like gnome tweak tool and gnome extensions, as well as some additional features such as auto dark theme and window tiling.

Access to different tools and some additional settings

Pop-shell window tiling

You may have spotted in the previous section that the settings section of the layout switcher has an item called "Window Tiling (Pop-shell)".

This, in my opinion is a really excellent addition. Pop shell is essentially a window tiling system developed by System76 for PopOS.

If this feature is turned on you then have access to Pop-shell within Manjaro! The way it is integrated is by the addition of a simple icon in the top bar.

System76 Pop shell in Manjaro

This allows you to quickly and easily have an easy to use window tiling system when it suits the situation, and then turn it off when you don't need it. All integrated perfectly and easy to use.

Change your kernel on demand

Not something that I specifically use, but is noteworthy, is that Manjaro has within it's GUI settings manager a kernel manager.

Manjaro kernel manager

This means you can easily switch kernel at a moments notice. I imagine some people will find this quite useful.

Conclusion

All in all Manjaro is really well rounded OS.

plenty of features to allow you to easily set up the OS the way you want it,
solid and well thought out packageing system and software so you don't end up with an unkempt mess after a few months of use.
no major updates to contend with every six months
looks pretty good out of the box

I recommend you give it a try.

How to Use a Yubikey in WSL2 on Windows

Wed, 22 Dec 2021 00:00:00 GMT

Windows now includes, natively, a subsystem that provides access to a real instance of Linux (by default Ubuntu). Access to a bash commandline, and Linux tools, can make some situations much easier. However, if you use a yubikey, or other hardware based authentication, it is not obvious how to utilise these within the Linux subsystem for ssh access to remote servers or github commits

This article is aimed at helping to get you setup in WSL2 with working yubikey gpg/ssh access.

Before we get started

It is assumed that you already have a yubikey, and that you have your gpg/pgp keys setup and ready to go.

If this is not the case, there are a plethora of tutorials available to guide you through it. Please set it up and come back here once you are ready.

The yubikey must function for GPG and SSH in Windows

Before we dive into the WSL2 environment, it is probably wise to check that the yubikey works in a Windows
environment as we would expect. Essentially, we are just creating a path for the yubikey to access authentication tools from Windows...so if your Yubikey doesn't work properly in Windows, it won't in WSL2 either.

If you are happy that all is well in Windows, then you can skip this section, otherwise make sure to follow the steps that follow.

Install GnuPG and PuTTY in Windows

You should install GnuPG and PuTTY.

This will give you all the tools you need to successfully use a yubikey in Windows for SSH.

As part of the install for GnuPG you should have a program called Kleopatra. If you open this program and plug in your yubikey, you should be able to click on "Smartcards" in the interface, then click F5 on your keyboard, and it will display the info about the yubikey.

GPG Config file

To enable SSH to work properly you will need to create a config file.

If we assume our username is "Dave" then the path to this file would be:

C:\Users\Dave\AppData\Roaming\gnupg\gpg-agent.conf

(Obviously change "Dave" to your own username on Windows in the path above)

The above mentioned file will likely not exist. If it doesn't, then just create a blank file, otherwise use the one that is already there.

Then you need to add to the file:

enable-putty-support
enable-ssh-support
default-cache-ttl 600
max-cache-ttl 7200

Ultimate trust for your public key

Although potentially not essential to this process, it is probably a good idea to include this step.

Essentially we are going to give our public key ultimate trust. I am going to assume you can locate your own public key, from either your own location, or from a keyserver. Once you have located it you need it import it into Kleopatra, or on the commandline if you know how.

Now we are going to assign the key ultimate trust.

Open powershell and type gpg --card-status and you should see various information about your yubikey. If not, something is not working correctly, try rebooting and give it another go. If you still have issues try going through the steps above again.

The next command is gpg --list-keys. This will give you the gpg public key. The top two lines of the output will look something like this:

pub   rsa4096 2019-05-02 [SC]
  1CA87B39873495770098080098336BC4E5C445AB

The long string is your public key fingerprint, which you will need in the next step.

Type gpg --edit-key 1CA87B39873495770098080098336BC4E5C445AB this will bring up a gpg> prompt in which you should type "trust". It will then prompt you for a number. Enter "5", which is ultimate trust. Then confirm as instructed.

If you now go and check in Kleopatra or type gpg --list-keys in the commandline you will see your key has been assigned ultimate trust.

Final checks

You should now be in a position where SSH works within windows.

Before we proceed, it would make sense to check you can actually use SSH within windows with your yubikey. Either use PuTTY to SSH into a server, or try to sign a git commit etc., and you should see that it is fully functional.

If this is not the case, then something is amiss, and there is no point proceeding to the next stage as it will not work.

One thing you can try is running in powershell the following:

gpgconf --kill gpg-agent
gpg-connect-agent /bye

This should restart gpg-agent so you can give it another shot.

WSL2 Setup

Install WSL2 (Ubuntu)

Firstly, install WSL2, which is as easy as running the following command in a powershell prompt with administrator privileges:

wsl --install

It will take you through the various install steps, restarts etc. It is very straight forward. You can install other distributions than Ubuntu, but by default Ubuntu is installed.

After the install you can search for Ubuntu in Windows search, and it will open for you.

If you ever need to properly close a WSL instance, type the following command into the terminal:

wsl.exe --shutdown

Just closing the window will not do this.

Initial steps

Make sure you are up to date:

sudo apt update
sudo apt upgrade

Run the following commands (change the wsl2-ssh-pageant version number in the download link as appropriate):

sudo apt install socat
mkdir ~/.ssh
wget https://github.com/BlackReloaded/wsl2-ssh-pageant/releases/download/v1.4.0/wsl2-ssh-pageant.exe -O ~/.ssh/wsl2-ssh-pageant.exe
chmod +x ~/.ssh/wsl2-ssh-pageant.exe

Edit bash-rc

Append the following to your ~/.bashrc file (use nano, vim etc.):

# SSH Socket
# Removing Linux SSH socket and replacing it by link to wsl2-ssh-pageant socket
export SSH_AUTH_SOCK="$HOME/.ssh/agent.sock"
if ! ss -a | grep -q "$SSH_AUTH_SOCK"; then
  rm -f "$SSH_AUTH_SOCK"
  wsl2_ssh_pageant_bin="$HOME/.ssh/wsl2-ssh-pageant.exe"
  if test -x "$wsl2_ssh_pageant_bin"; then
    (setsid nohup socat UNIX-LISTEN:"$SSH_AUTH_SOCK,fork" EXEC:"$wsl2_ssh_pageant_bin">/dev/null 2>&1 &)
  else
    echo >&2 "WARNING: $wsl2_ssh_pageant_bin is not executable."
  fi
  unset wsl2_ssh_pageant_bin
fi
# GPG Socket
# Removing Linux GPG Agent socket and replacing it by link to wsl2-ssh-pageant GPG socket
# NOTE: the "config_path" and "-gpgConfigBasepath ${config_path}" are generally needed for
# gpg4win 4.0 and later 
export GPG_AGENT_SOCK="$HOME/.gnupg/S.gpg-agent"
if ! ss -a | grep -q "$GPG_AGENT_SOCK"; then
  rm -rf "$GPG_AGENT_SOCK"
  wsl2_ssh_pageant_bin="$HOME/.ssh/wsl2-ssh-pageant.exe"
  config_path="C\:/Users/<username>/AppData/Local/gnupg" # replace <username> with your own username
  if test -x "$wsl2_ssh_pageant_bin"; then
    (setsid nohup socat UNIX-LISTEN:"$GPG_AGENT_SOCK,fork" EXEC:"$wsl2_ssh_pageant_bin -gpgConfigBasepath ${config_path} --gpg S.gpg-agent" >/dev/null 2>&1 &)
  else
    echo >&2 "WARNING: $wsl2_ssh_pageant_bin is not executable."
  fi
  unset wsl2_ssh_pageant_bin
fi

Now restart WSL2:

wsl.exe --shutdown

You should now have access to your Yubikey in WSL2:

Try gpg --card-status in WSL2 and you should get info about your yubikey. If not something is wrong. Try going through the steps again.

It should also be possible to use SSH. For example ssh-add -L should return a key.

Change key trustworthiness to ultimate

This is the same thing we did previously in windows, except we will do it on the commandline (obviously use your own key):

$ gpg --edit-key 1CA87B39873495770098080098336BC4E5C445AB
gpg> trust
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu

Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y
gpg> quit

If you now run gpg --list-keys the keys should have ultimate trust.

Done!

That should be it. You should be able to SSH into a server or use git commits etc.

Visual Studio Code

One of the things you may want to do is use a text editor/IDE rather than using VIM or Nano on the commandline. This is certainly preferable if you are a software developer and need to edit lots of text files.

Fortunately, you can easily use a program such as VS Code for this.

How to use VS Code in WSL2

You do not need to install VS Code within the WSL2 Ubuntu install. You only need VS Code installed in windows as normal.

On the commandline in WSL2 just type the following:

code .

The dot is quite important so don't forget that!

It will then open a VS Code instance with the path of where you are in the terminal, so maybe "cd" to your project root path first before running it.

It is quite well intergrated so if for example you are running a python virtual environment, it will pick this up and allow you to select the virtual environment that exists within Ubuntu. This greatly helps with general work, and debugging.

References

The main place that the info in this article comes from is here:

Jaroslav Živný

You can also refer to the information provided by the wsl2-ssh-pageant tools author on github:

wsl2-ssh-pageant

The Power of the Continuous Wavelet Transform (CWT) in Machine Learning

Fri, 01 Jul 2022 01:00:00 GMT

The purpose of this article is to look into whether the use of the Continuous Wavelet Transform (CWT) is beneficial as a preprocessing technique before utilising a neural network to predict a human gesture classification problem.

There will be no heavy mathematics, with the focus being on the implementation, and results. Where necessary some explanation will be provided as to why certain parameters are used.

Why use the CWT?

One of the main challenges with machine learning is feeding in data that is going to be easy for the model to interpret and learn from, without losing any important information from the original data. If you can achieve this, it allows for simpler lightweight models, and reduces the chances of, for example, over-fitting to noise or other anomalies in your raw data.

The CWT can potentially help you achieve this.

The basics

The CWT is a signal processing technique similar to the Fourier Transform, in that it lets you extract and separate out frequency information from a timeseries. Where it differs from the Fourier Transform is that it can also retain the time domain information as well (i.e. it can display the frequency data and where it occurred along the timeseries).

The output for a single 2D input time series (x: time, y: amplitude) is therefore a 3D output matrix (x: time, y: frequency, z: amplitude). This means the output of a CWT can be rendered as a pictogram (i.e. visualised in a 2D image rather than a line).

Where is the advantage?

We have turned the original signal into a picture, which is great for humans, as it is more interpretable for us. However, we are going to feed this into a neural network, so if it contains the same information as the original signal why bother?

Turns out you have a few dials to play with when tuning the transform that will differentiate your CWT output from the original signal.

Choosing your wavelet

There are predefined wavelets that you can use against the signal. Each wavelet is different in terms of shape and characteristics, you should pick a wavelet shape that fits well with the characteristics of the signal you are trying to process. I won't go further into this selection process here, but if unsure the Morlet wavelet is a good starting point, and what we will use throughout this article.

The morlet wavelet

The wavelet is basically convolved (multiplied) with the time series at each timestep, moving across the signal. This is done at varying 'scales', so scale 1 will be for high frequencies (i.e. the wavelet is narrow and picks up higher frequencies in the signal). As you increase the scale the wavelet is stretched horizontally, and therefore a better match to lower frequencies. This process is repeated for a variety of scales which allows the extraction of frequency information from the signal.

Setting the scale

As mentioned in the previous section you can stretch the signal using a parameter called 'scale'. This is equivalent of targeting specific frequencies within the signal.

You can literally specify a range of scales to be processed, so this gives you the flexibility to, for example, filter out high frequency noise from the signal. Or very specifically target a certain frequency range.

Use as an exploratory tool

As mentioned earlier you can make visualisations from the CWT, so one approach might be to visually review a random selection of data to help determine where the most appropriate 'scale' range is, and fine tune the scale to that range before passing to the neural network.

Complex wavelets

Some of the wavelet transforms are complex (i.e. the wavelets extend into the complex domain). Ultimately, this means that you can extract phase information from a signal, if this is of use for your analysis. However, it is worth noting that this is an additional layer of information that may be useful for a neural network. We will utilise complex wavelets (in a very superficial way) in this article.

Why not use the Discrete Wavelet Transform (DWT)?

You may have heard of the Discrete Wavelet Transform (DWT), and wonder why we don't use that?

The truth is they are very similar (both being wavelet transforms), however to get into the differences at any meaningful level requires a dive into mathematics, which is not what this article is for.

Very simply the DWT also breaks down the signal into frequency components. However, each 'level' of detail (like the scales in the CWT) extracted from the original time series results in a halving of the samples (i.e. the signal gets shorter). This is more efficient computationally than the CWT, but not ideal for our purposes here. You can't produce an image for example.

What the DWT is very useful for is filtering. As an example, if you want to filter noise from your signal the DWT is an excellent choice, but that is beyond the remit of this article.

The data

Direct quote from the data source:

Eight different users performed twenty repetitions of twenty different gestures, for a total of 3200 sequences. Each sequence contains acceleration data from the 3-axis accelerometer of a first generation Sony SmartWatch™, as well as timestamps from the different clock sources available on an Android device. The smartwatch was worn on the user's right wrist. The gestures have been manually segmented by the users performing them by tapping the smartwatch screen at the beginning and at the end of every repetition.

An example of each of the movements performed by the participants and their associated labels can be seen in the image below:

The movements performed with the watch and associated labels

Source

Source dataset direct link

The plan

This is the overall plan of how the data will be prepared and compared.

If you wish to look in detail at the code used to produce the results that follow, please feel free to reference the jupyter notebook, which is available on my github here:

Jupyter Notebook

Data split

Initially, I will use 7 out of the 8 people as training and validation, and the 8th person as a holdout test set.

The 7 people will be completely randomised and then split 85%-15% (train-validation). The final outcome of the models being judged on the holdout test 'person'.

This should result in:

training timeseries --> 2380 sequences
validation timeseries --> 420 sequences
testing timeseries --> 400 sequences

Total timeseries: 3200

Models

The models that will be created are as follows:

Model 1 - A CNN model (Conv1D) used as a baseline on the raw timeseries data - this is the benchmark
Model 2 - A CNN model (Conv2D) utilising a CWT on the timeseries before input into the model
Model 3 - A CNN model (Conv2D) utilising a complex CWT on the timeseries before input into the model

All the above models will use the same parameters and number/type of layers to keep them as comparable as possible.

Comparison

Stage 1

A single run of the model to get an idea of accuracy and see where the model is failing (or succeeding) to generalise.

(Users 1 to 7 as train/validation, User 8 as holdout test).

Stage 2

Ten repeat runs to get a more accurate average accuracy, which removes any variation due to numerical randomness / initialisation parameters.

(Users 1 to 7 as train/validation, User 8 as holdout test).

Stage 3

For Model 1 and either Model 2 or 3 (depending which performs best), a cross validation of users will be performed.

Essentially, each individial user (1 to 8) will be used as the hold out test set in a completely independent set of tests. Each set of tests will be repeated 10 times (like Stage 2) to get an average accuracy.

This will give a good indication as to how the models perform for each individual, ultimately giving a better indication as to how the model will likely perform with a completely new user in the future.

Preprocessing the data

Exploration

In an initial exploration of the data the following points were discovered:

the data is sampled at around 9Hz (0.11s per sample)
the timeseries samples are all of different lengths (longest 51 sample (5.61s), average of 20 samples (2.2s))
the total amount of timeseries is not 3200 it is actually 3251, with some users having more samples than others (although still a very even split, not highly skewed to one user or another)
the accelerometer data is close to normally distributed for the purposes of scaling
accelerometer in the z direction looks to have a more significant mean offset from zero than the other two components (possibly gravity?)

An example of all the movements performed by User 1

Preprocessing

One of the initial problems with the dataset is that the timeseries are all of different lengths. This means we need to pad the sequences to length of the longest sequence (assuming we don't want to truncate the data).

We will take two different approaches here. The first will be for the raw timeseries data and the second for the data intended for the CWT.

As the data crosses zero we will use a large padding value (-9999.0) on the timeseries input data, and then use a masking layer to mask these values in the model. As the CWT is a pre-processor before the model, we cannot feed exaggerated padding values like this into the CWT, as it will heavily skew the data. A zero pad will therefore be used on the data that will be fed into the CWT.

To ensure that the zero pad will not skew the data too much SciKitLearn's StandardScaler will be used to scale the data to zero mean and unit variance before applying the pad (we have already confirmed the data is relatively normally distributed so this should be appropriate). For consistency, and the benefits scaling generally provides for neural networks anyway, this scaling will also be applied to the raw time series data used in the reference model (Model 1).

An example of data after passing through StandardScaler (before padding)

All models will also include a scaling layer to scale the input data between the values of 1 and -1 before hitting the neural network. The scaling value used will be based on the highest absolute value across all accelerometers after standard scaling (rather than each individually) to retain relative magnitude between the sensors.

Note: the data used to fit StandardScaler consisted of the whole dataset. Only the training dataset should really be used for this otherwise you leak information about the validation and test sets, which are meant to be independent. In this particular case it is not a big deal, as we are just exploring, but if you have to present reliable bulletproof figures, please do not do this.

Picking scales for the CWT

One of the items that first needs to be considered before jumping into preprocessing the data with a CWT is what scales you wish to compute the continuous wavelet transform over, as this will decide how well the signal is represented in the output.

The scales are arbitrary, but a sensible selection can be made by translating the scales into associated frequencies. Scales can be 'translated' into frequencies if the sample rate of the data is known. Basically, a small scale (1 for example) is related to a higher frequency, and a larger scale to a lower frequency. This is basically due to the increase in scale 'stretching' the wavelet, and hence being a better match to 'longer' signals (i.e. lower frequencies).

In our case the sample rate of the data is approximately 9Hz (0.11 seconds, sample to sample). This is not a particularly high sample rate, so we need to retain as much of the data as possible.

Unfortunately, the CWT is subject to the Nyquist frequency, so in theory any frequency above 4.5Hz will experience aliasing, which is not ideal as it will polute the signal.

At the other end of the scale:

Our longest signal is 51 timesteps long which is 5.61s (which is about 0.18Hz). The average signal is 20 timesteps long which is 2.20s (which is about 0.45Hz)

This gives a good starting point for picking our scales. To keep the output small, as we are not using particularly deep neural networks, we will limit the lower frequency to about half the maximum, so ~0.36Hz and try to get as close to 4.5Hz as we can. This should give us a nice range to work with.

Here is the code which shows the result of the above. The output is an array of frequencies hitting the range we discussed above:

# Standard Morlet Wavelet frequencies at double the sampling frequency
dt = 0.11  # ~9 Hz sampling
input_scales = np.arange(2,22, dtype='float32')
input_scales = np.insert(input_scales,0,1.64)
frequencies = pywt.scale2frequency('morl', input_scales) / dt
frequencies

array([4.5038805 , 3.6931818 , 2.4621212 , 1.8465909 , 1.4772726 ,
       1.2310606 , 1.0551947 , 0.92329544, 0.8207071 , 0.7386363 ,
       0.67148757, 0.6155303 , 0.5681818 , 0.52759737, 0.49242425,
       0.46164772, 0.434492  , 0.41035354, 0.38875598, 0.36931816,
       0.35173163], dtype=float32)

# Complex Morlet Wavelet frequencies at double the sampling frequency
input_scales_comp = np.arange(2,27, dtype='float32')
frequencies = pywt.scale2frequency('cmor1.5-1.0', input_scales_comp) / dt
frequencies

array([4.5454545 , 3.0303032 , 2.2727273 , 1.8181819 , 1.5151516 ,
       1.2987014 , 1.1363636 , 1.0101011 , 0.90909094, 0.8264463 ,
       0.7575758 , 0.6993007 , 0.6493507 , 0.6060606 , 0.5681818 ,
       0.53475934, 0.50505054, 0.4784689 , 0.45454547, 0.43290043,
       0.41322315, 0.39525694, 0.3787879 , 0.36363634, 0.34965035],
      dtype=float32)

So from the above we can see that the appropriate scales to hit our intended frequency ranges for the data will be:

Morlet: 1.64 to 21
Complex morlet: 2 to 26

Example complex wavelet graph outputs

Note: In the example above there are six graphs as the complex wavelet has both imaginary and real parts. The normal CWT will produce only three graphs, as there are no imaginary parts.

What do the graphs show?

The graphs above are printed in grey and red fading to white at the centre value (roughly zero in this case). I have left a scale bar off as it is mostly irrelevant due to the data being scaled. So when you see dark red or dark grey, those are high energy areas (peaks or troughs). This is where most of the information resides in our signal.

You can immediately see both in terms of time and scale (or frequency) where in the signal has the most information. This could help you further tune the scales to focus on a particular area of interest, should you want to experiment further.

The models

The models were constructed to be as close as possible, and simple. I have included dropout and pooling layers in the model to reduce overfitting since the data is limited and not particularly complicated.

# Model 1 - for raw timeseries data

    if model_number == 1:
        model = Sequential([
                        Input(shape=input_shape),
                        Masking(mask_value=-9999.0),
                        Rescaling(scaling_value),
                        Conv1D(filters=64,
                               kernel_size=4,
                               strides=1,
                               padding='valid',
                               kernel_initializer='glorot_uniform',
                               activation='relu'),
                        MaxPooling1D(),
                        Dropout(0.2),
                        Conv1D(filters=32,
                               kernel_size=1,
                               strides=1,
                               padding='valid',
                               kernel_initializer='glorot_uniform',
                               activation='relu'),
                        MaxPooling1D(),
                        Flatten(),
                        Dense(64, activation='relu'),
                        Dropout(0.2),
                        Dense(20,activation='softmax')
        ],name='Conv1D_Model_1')

    # Model 2 - for CWT images 
    # (Model 3 (for the complex CWT) is the same as this, but has been cut for brevity)

    elif model_number == 2:
        model = Sequential([
                    Input(shape=input_shape),
                    Rescaling(scaling_value),
                    Conv2D(filters=64,
                           kernel_size=4,
                           strides=1,
                           padding='valid',
                           kernel_initializer='glorot_uniform',
                           activation='relu'),
                    MaxPooling2D(),
                    Dropout(0.2),
                    Conv2D(filters=32,
                           kernel_size=1,
                           strides=1,
                           padding='valid',
                           kernel_initializer='glorot_uniform',
                           activation='relu'),
                    MaxPooling2D(),
                    Flatten(),
                    Dense(64, activation='relu'),
                    Dropout(0.2),
                    Dense(20,activation='softmax')
        ],name='CWT_Model_2')

The models are all compiled with the Adam optimiser and run for 50 epochs each. At the end of the 50 epochs the best weights are restored based on val_loss.

Learning rates were tuned once for each model prior to the main runs:

Model 1 - 0.005
Model 2 - 0.001
Model 3 - 0.001

On the main runs a scheduler also halved the learning rate every 20 epochs to help the model converge.

The results

All models managed an accuracy and val_accuracy of 99%. Game over? Not really...

Model 2 - Accuracy training curve

What that means is that our models have all learnt really well (or over-fit). Remember, our test and validation set is a random mixture of 7 users, but the hold out test set is a completely different person that doesn't exist at all in training or validation sets. So for the trained models on the initial run on the holdout test set (User 8):

Model	Accuracy	F1-Score
1 (Raw time-series)	0.88	0.88
2 (CWT)	0.93	0.92
3 (Complex CWT)	0.96	0.96

For an initial run that result looks pretty impressive! We still need to repeat the runs 10 times like we said at the start to remove any initialisation randomness and see how stable the outputs are, but it is looking promising.

What might be interesting is to see where the models are making mistakes...

The movements performed with the watch and associated labels

Model 1 - Confusion Matrix

Model 2 - Confusion Matrix

Model 3 - Confusion Matrix

As you can see by comparing the movements to Model 1 and 3, the movements that the models get wrong make a lot of sense (i.e. they are similar movements). The worst mismatched predicted labels are:

Model 1 - 16 confused with 12
Model 2 - 16 confused with 12
Model 3 - 20 confused with 19

Interestingly, although Model 3 on the whole does a better job, Model 1 manages to almost perfectly classify label 20 whereas Model 3 struggles with this particular label. It shows that the information is there, but Model 3 fails to capture it, or it has been removed by the transform. To further solidify this point, Model 2 also performs well for label 20, so the CWT is capable of capturing the correct information, but some fine tuning of scales is probably required.

Solidifying the numbers

To get a more solid grasp on the how the models perform the tests were repeated 10 times and the result averaged:

Model	Accuracy
1 (Raw timeseries)	0.859
2 (CWT)	0.945
3 (Complex CWT)	0.952

A box plot as another visual aid as to the results distribution:

Boxplot of the results distribution

The result of this is that there is a fairly significant advantage to using a CWT on this data (at least under the parameters used in this article). It is also noted that, although the difference is small, it may be beneficial to consider a complex wavelet transform to try and extract as much data as possible out of the time series before processing.

The definitive check - cross validation

In the previous sections we have explored the benefit of using the continuous wavelet transform in a neural network. So far the hold out test set has been user 8. However, we have no idea whether this user is a good representation of the general population, or a fairly unique individual.

A cross validation across all users will therefore be run (i.e. each user will be the hold out test set for it's own set of train/val/test runs). This should give a much more reliable indication of the performance that has been achieved, especially considering the small size of the dataset.

To avoid any bias due to randomness, we will also repeat each test 10 times and take an average, as we have done previously.

Mean and standard deviation cross-validation results

It seems the hold out user really does matter! Although there are a wide variation of results, depending on the user, almost across the board the CWT transform outperforms the normal time series model (from 2% to 15% improvement). The exception, as you can see, is User 4, where the CWT is beaten on average by the normal model. Although it should be noted that there is only a 2% difference, and Model 3 managed a quite respectful 92% accuracy in this case.

Note: the standard deviation graph presented is calculated across the 10 repeat runs for each user, and represents how consistent the model is across the 10 repeat runs.

Further to the above, it should be noted that the validation results across all tests and all users was very high (97%+).

Mean and standard deviation cross-validation results (validation)

This indicates that although the models were able to learn the data provided to them equally well, the CWT model was able to both, pick out more relevant features, and generalise better than than the normal time series model.

Conclusion

In this particular case I think we can conclude that the CWT should definitely be considered as a tool in the arsenal of machine learning practitioners. It will not be suitable for every case, perhaps because of the additional overhead of processing that is required. However, it is a flexible tool that can be moulded to suit your specific data, and potentially improve model bias and accuracy.

This article has only really touched the surface, as no in depth tuning of parameters has been performed. Items like the complex parameters of the complex Morlet wavelet were fixed throughout this experiment, and plenty of experimentation of a suitable range of scales could be conducted, so there is plenty of scope for further investigation in to the CWTs use in machine learning.

References

From the authors of the dataset:

G. Costante, L. Porzi, O. Lanz, P. Valigi, E. Ricci, Personalizing a Smartwatch-based Gesture Interface With Transfer Learning, 22nd European Signal Processing Conference, EUSIPCO 2014
L. Porzi and S. Messelodi and C.M. Modena and E. Ricci: A Smart Watch-based Gesture Recognition System for Assisting People with Visual Impairments. ACM International Workshop on Interactive Multimedia on Mobile and Portable Devices - IMMPD, Barcelona, Spain, 2013

Other references and articles of interest:

How I Passed the TensorFlow Developer Certificate Exam

Fri, 05 Aug 2022 01:00:00 GMT

Having been through the process of preparing for, and passing, the TensorFlow Developer Certificate exam, I thought it might be useful to give other people an overview of what to expect, the best resources to use to prepare, what to watch out for in the exam, and advice on whether it would be useful for you in particular.

Note: this article will NOT reveal any questions / answers on the actual exam. Although this might disappoint some people, it is very important to uphold the integrity of the exam. This is to ensure that when you finally get your certificate, it has value.

What is the TensorFlow Developer Certificate?

The TensorFlow certificate website describes this quite succinctly:

This certificate in TensorFlow development is intended as a foundational certificate for students, developers, and data scientists who want to demonstrate practical machine learning skills through the building and training of models using TensorFlow.

The program consists of an assessment exam developed by the TensorFlow team. Developers who pass the exam can join our Certificate Network and display their certificate and badges on their resume, GitHub, and social media platforms including LinkedIn, making it easy to share their level of TensorFlow expertise with the world.

...so it is a practical exam that requires you to build actual TensorFlow models to predict outcomes (categorical and regression).

Frame of reference for my advice

Now you know what the certificate is, I think it is important to understand a little about my background and experience before I continue. This may help some people formulate a plan of attack for their preparation, and also judge how relevant my advice is to them.

On a day to day basis I am a 'traditional' engineer (i.e. Mechanical Engineer, Civil Engineer etc.), not a software developer or data scientist / data analyst.
I have experience with software development, and data analysis within my day to day work roles (a lot of traditional engineering work is data analysis of some sort).
I have made a concerted effort over the past 5 to 6 years to expand and improve upon my software development / data analysis skill set. This is both due to it being beneficial to my work, and also out of interest in the subject in general.
Although I have a Masters degree, it is not in the software development domain (Aeronautical Engineering).
I spend a reasonable amount of my spare time working on various software / analytics projects.

The above should give you a good gauge as to where your level of experience might fit compared to me, and allow you to take the advice that follows as such.

Is it worth doing?

This is a very valid question, and one I want to cover before I explain what resources I recommend etc.

Cost

It costs $100 to do the certificate. That might be fine for some people, but I can see this being a significant hurdle for some.

Fortunately, there is a stipend program if you meet the criteria, so this could be help to reduce this cost in some cases.

Job prospects

If the cost doesn't put you off, then the main question is:

Will this certificate improve my job prospects?

In my opinion, this all depends on your current status. If you have a CS degree and are currently employed as a data scientist with years of experience, then this is going to add little value to your CV. However, as you peel away the skills, the further up the pecking order this certificate will go:

Have a CS degree but no data science experience?
Have no CS degree, and some data analyst experience?
Have none of the above?
Straight out of university?

In all of the above situations (and many variations inbetween) it is clear to see that some value can be added by this certificate. It is a proof of competence in a very specific skill.

However, I think the most important factor to remember is visibility.

If the only thing this certificate achieves is getting your CV noticed initially, it is worth it.

When applying for jobs, the person that initially scans your CV (and they do just scan them) may be checking against key criteria, and if one of those is TensorFlow then you have a clear defining proof of competence. If the only thing this certificate achieves is getting your CV noticed initially, it is worth it. This could even apply to very experienced data scientists, as the person who initially scans the CVs at larger companies may not have significant domain knowledge.

So is it worth it? I think in most cases, yes. If you can stomach the cost.

Is it difficult, and what do I need to know?

The best place to start in terms of understanding what knowledge is required for the exam is the candidate handbook.

It details exactly which topics you will be expected to know in five major sections, and even breaks down each section into specific skills. The five sections are:

TensorFlow developer skills
Building and training neural network models using TensorFlow 2.x
Image classification
Natural language processing (NLP)
Time series, sequences and predictions

If you can go through this checklist and tick off most, if not all the requirements, then you should be in good stead for the exam. If not, then you need some prep work...

Photo by Siora Photography on Unsplash

we will get to that shortly!

Difficulty

Even if you have experience, or work as a data scientist, you will likely need to do some preparation for this exam (although there are always exceptions!). If you think you can do no prep and pass this exam, I think you might be in for a shock.

Is this because it is really hard? No. But it isn't a walk in the park either, you need to know more than the basics.

If you work as a data scientist you will likely have a very specific skill set (object detection / classification maybe?), and this test also covers Natural Language Processing (NLP), among others, which you may be rusty on. So you may just need a refresh.

To summarise, it is not something you can just jump into with zero prep. If you are relatively new to the subject you will need to study properly, and if you have experience you will likely need a bit of a refresh on some areas that you may not have used in a while.

How should I prepare?

Essentials

Before I get into the courses I recommend to prepare for the test, I just wanted to mention that I am assuming that you are familiar with the Python programming language. This is kind of essential. If not you should first start by learning Python.

Photo by David Clode on Unsplash

Furthermore, I would also recommend that you are familiar with the following, before you attempt this certificate:

Python libraries: pandas, numpy, matplotlib, scikit-learn.
Some practise with machine learning algorithms - this is not essential, but gives you a solid base, and typically provides exposure to the ins and outs of libraries such as scikit-learn without the added complication of TensorFlow.

Courses

There are some great courses designed to help you, and it is very easy to tune based on your experience level.

There are two courses I recommend, both are designed to help with this certification:

The two courses above are very different in their method, and I would recommend the approach detailed in the following sections based on your experience.

In all cases

I am about to recommend which resources I think are best in relation to your skill level. However, I need to ram home one particular item first:

Regardless of whether you use the courses I suggest, or books, or YouTube videos to prepare for the exam. It ESSENTIAL to practise by coding it out yourself. Just watching the videos / reading will not be enough in most cases.

Experience: mid-level to expert

If are reasonably familiar with TensorFlow and comfortable building, testing and debugging models, then I would recommend you go with the course from Coursera.

The course is listed as taking "Approximately 4 months to complete", but this is not really true. You could easily get through the course content in a couple of days if you already have experience with the subject matter.

Content is concise and to the point, with plenty of explanation of why things are done a certain way. You also have access to coded examples and walkthroughs of code.

It covers all the main areas that are required for the exam.

It should also be noted that this course is an officially listed course on the TensorFlow website.

Experience: Little to none

If you have very little experience with TensorFlow or machine/deep learning in general, I would recommend that you start with the course from Udemy.

While the Coursera course is concise and focused, the course from Udemy is significantly more drawn out. It will take quite a while to get through all the videos, but if you stick with it you will have a great general understanding of the subject, rather than just hyper-focused content for an exam.

The course is taught in a very code-centric way, and the teacher codes throughout the videos. What is interesting is the teacher makes mistakes (most likely deliberately) as they are coding, and teaches you approaches to fix things on the fly. The mistakes are usually things that are overlooked by beginners, so it is a very useful feature of the videos.

However, if you have experience you may find these videos repetitive and annoying after a while, hence why I think the Coursera course is preferable if you have a bit more experience.

I should also note that the Jupyter notebooks provided for this course are extensive, and a very useful reference.

Once you have finished the Udemy course, I still HIGHLY recommend that you go through the Coursera course. It covers some content you won't see in the Udemy course, and is a great general refresher that can be completed relatively quickly.

Other courses

There is also a course available from Udacity, but I have not seen the content, so cannot comment on whether it is a good course or not. Might be worth checking out though.

What is the exam like?

The basics:

it is open book, so you can use all the resources you need throughout the exam
it is timed (5 hours). You can finish at any time, but the test will auto submit if 5 hours elapses.
you are tested on the models. Models are uploaded by you, and a score out of 5 is returned. You can submit as many models as you want over the course of the test, but you must submit only one final model for each section.
you must do the test in PyCharm IDE with a specific version of TensorFlow and Python.

Photo by Bima Rahmanda on Unsplash

Setup

You have to use a specific version of PyCharm to complete the test, you then install a specific 'TensorFlow Test' Module. This module creates the test environment and provides the interface and buttons you need to complete and submit the test.

When you 'buy' the test you will be sent a pdf which details exactly what you need to setup, down to the version of PyCharm and Python that you must use, and the packages that it will install in the environment.

You can check out the latest environment setup guidance here.

I would highly recommend setting up a test environment in PyCharm using the exact version of Python and packages that the guidance details. This should mitigate any potential problems you might come across in the exam. This is important, as the test is timed (5 hours), so you don't want to waste time debugging your environment while the test clock is running!

Starting the test

Once you are setup and you have clicked to 'start the test' you will have folders / files that contain everything you need to complete each section of the test. Instructions on exactly what you need to do are contained within the files that you will write the code. Obviously I can't go into detail here, but once you are at this point you just read the instructions and go!

Grading

You are graded on the models that you generate. These models are uploaded using the plugin (you just hit a button) and then the model you submit is tested. Throughout the course of the test you can submit as many models as you like, so you have plenty of opportunity to improve your models throughout the course of the exam.

When you submit a model you receive a grade out of 5 for the submitted model. For example, you may submit a model and the result comes back as 3/5. You might therefore want to try and improve the model to get 5/5 before you end the test, or the time runs out.

I should point out at this point that I have no idea what you need to achieve in terms of test scores to ensure you pass. I completed the test with 5/5 in every section, but this may not be necessary. I have no way of knowing.

Recommendations

One of the things to remember is that this is an open book exam. You can use any resources you want to help you as you go through the exam, so have the resources you think will be helpful setup and readily accessible.

If you do not have a GPU in the computer that you will use for the test, then you need to bear in mind that some of your models (depending on your strategy and choices in the test) may take a little while to run through. In reality the test time of 5 hours takes this into account, and the models you are required to build are reasonable, so it is not a major problem. However, if possible using a GPU would be preferable, and save you some time.

One good option, if GPU access is not possible, is to use Google Colab to run your tests, and then transfer your code to PyCharm on your final model run. This is perfectly valid and even mentioned in 'setting up' documentation:

During the exam, you are welcome to experiment with training models using GCP, AWS, Jupyter Notebooks or Google Colab, but you will still need to define, train and save your models within the exam environment, inside PyCharm, in order to submit it for grading

This allows you to take advantage of the free GPU provided in a Colab environment, which should help you iterate a bit quicker as you fine tune your models.

Problems

In general the test went very smoothly. However, I did have an issue with my final model. When I ran the code for the model within PyCharm it would not save the model, and instead gave an error.

Photo by Elisa Ventur on Unsplash

In the end I had to run the code in Colab and transfer the model to PyCharm to upload it. Fortunately, that worked fine, but due to the requirements of a specific version of Python etc. I can't guarantee this will always work. So, where possible I would try to stick to running your models in the PyCharm environment. It is, however, good to know that there is a potential alternative should you hit a technical issue like I did.

How do I know if I passed?

Once you have submitted your test with your final models, you should receive a confirmation that you completed the test.

In my case, shortly after that I received an email saying I had passed. Literally within 5 minutes or so after completion. Again, I can't guarantee this will happen in your case, but that is what happened to me.

What do you get in the end?

Certificate

You are basically issued with a certificate through Google's partner service accredible:

TensorFlow Certificate

Through the interface you have on your Accredible account you can do various things like download the certificate, create Gmail signatures, outlook signatures, and various embedding options for webpages.

The certificate is valid for 3 years, after which:

To renew your certification at that time, you need to complete the registration and certificate process again.

Slightly vague wording, but I think that means you need to do the exam again.

TensorFlow Certificate Network

Apart from the actual certificate, you are also added to the TensorFlow Certificate Network (if you opt in).

This is basically an online searchable directory of all the people that have passed the TensorFlow certification. You can filter by name, country, experience etc. so it is very useful for employers, or potential clients to find local 'experts'.

TensorFlow Certificate Network

Again, this may or may not add value for you personally, it is impossible to say, but it is extra visibility none the less.

Conclusion

All in all I think the TensorFlow Developer Certificate is a positive addition to the machine learning / deep learning landscape.

It has spawned a multitude of online courses that are tuned to give you a broad understanding of the main problems (Natural Language Processing (NLP), image classification, time series prediction etc.) that are currently being used within industry to solve a multitude of real world problems.

This gives beginners a structured way to get a good feel for the machine learning / deep learning topic. It also allows more experienced practitioners to check or level-up their skill set. Even seasoned veterans of the industry may benefit from the added benefit of having a certification from Google, one of the biggest players in the deep learning space.

Is it essential to do it? Of course not, but depending on your circumstances it could be of real benefit.

How to Speedup Data Processing with Numpy Vectorization

Thu, 25 Aug 2022 01:00:00 GMT

When dealing with smaller datasets it is easy to assume that normal Python methods are quick enough to process data. However, with the increase in the volume of data produced, and generally available for analysis, it is becoming more important than ever to optimise code to be as fast as possible.

We will therefore look into how using vectorization, and the numpy library, can help you speed up numerical data processing.

Why is python slow?

Python is well known as an excellent language for data processing and exploration. The main attraction is that it is a high level language, so it is easy and intuitive to understand and learn, and quick to write and iterate. All the features you would want if your focus is data analysis / processing and not writing mountains of code.

However, this ease of use comes with a downside. It is much slower to process calculations when compared to lower level languages such as C.

Photo by Wolfgang Hasselmann on Unsplash

Fortunately, as python is one of the chosen languages of the data analysis and data science communities (among many others), there are extensive libraries and tools available to mitigate the inherent 'slowness' of python when it comes to processing large amounts of data.

What exactly is vectorization?

You will often see the term "vectorization" when talking about speeding calculations up with numpy. Numpy even has a method called "vectorize", as we will see later.

A general Google search will result in a whole lot of confusing and contradictory information about what vectorization actually is, or just generalised statements that don't tell you a great deal:

The concept of vectorized operations on NumPy allows the use of more optimal and pre-compiled functions and mathematical operations on NumPy array objects and data sequences. The Output and Operations will speed up when compared to simple non-vectorized operations.

- GeekForGeeks.org - the first google result when searching - what is numpy vectorization?

It just doesn't say much more than: it will get faster due to optimisations. What optimisations?

What optimisations?

The trouble is that numpy is a very powerful, optimised tool. When implementing something like vectorization, the implementation in numpy includes a lot of well thought out optimisations, on top of just plain old vectorization. I think this is where a lot of the confusion comes from, and breaking down what is going on (to some degree at least) would help to make things clearer.

Breaking down vectorization in numpy

The subsequent sections will breakdown what is typically included under the generalised "vectorization" umbrella as used in the numpy library.

Knowing what each does, and how it contributes to the speed of numpy "vectorized" operations, should hopefully help with any confusion.

Actual vectorization

Vectorization is a term used outside of numpy, and in very basic terms is parallelisation of calculations.

If you have a 1D array (or vector as they are also known):

[1, 2, 3, 4]

...and multiply each element in that vector by the scalar value 2, you end up with:

[2, 4, 6, 8]

In normal python this would be done element by element using something like a for loop, so four calculations one after the other. If each calculation takes 1 second, that would be 4 seconds to complete the calculation and issue the result.

However, numpy will actually multiply two vectors together [2, 2, 2, 2] and [2, 4, 6, 8] (numpy 'stretches' the scalar value 2 into a vector using something called broadcasting, see the next section for more on that). Each of the four separate calculations is done all at once in parallel. So in terms of time, the calculation is completed in 1 second (each calculation takes 1 second, but they are all completed at the same time).

A four fold improvement in speed just through 'vectorization' of the calculation (or, if you like, a form of parallel processing). Please bare in mind that the example I have given is very simplified, but it does help to illustrate what is going on on a basic level.

You can see how this could equate to a very large difference if you are dealing with datasets with thousands, if not millions, of elements.

Just be aware, the parallelisation is not unlimited, and dependent on hardware to some degree. Numpy is not able to parallelise 100 million calculations all together, but it can reduce the amount of serial calculations required by a significant amount, especially when dealing with a large amount of data.

If you want a more detailed explanation, then I recommend this stackoverflow post, which does a great job of explaining in more detail. If you want even more detail then this article, and this article are excellent.

Broadcasting

Broadcasting is a feature of numpy that enables mathematical operations to be carried out between arrays of different sizes. We actually did just that in the previous section.

The scalar value 2 was "stretched" into an array full of 2s. That is broadcasting, and is one of the ways in which numpy prepares data for much more efficient calculations. However, saying "it just creates an array of 2s" is a gross oversimplification, but it is not worth getting into the detail here.

Numpy's own documentation is actually quite clear here:

The term broadcasting describes how NumPy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes. Broadcasting provides a means of vectorizing array operations so that looping occurs in C instead of Python.

- numpy.org

A faster language

Photo by Uillian Vargas on Unsplash

As detailed in the quote from numpy's own documentation in the previous section, numpy uses pre-compiled and optimised C functions to execute calculations.

As C is a lower level language, there is much more scope for optimisation of calculations. This is not something you need to think about, as the numpy library does that for you, but it is something you benefit from.

Homogeneous data types

In python you have the flexibility to specify lists with a mixture of different datatypes (strings, ints, floats etc.). When dealing with data in numpy, the data is homogeneous (i.e. all the same type). This helps speed up calculations as the data type does not need to be figured out on the fly like in a python list.

This can of course also been seen as a limitation, as it makes working with mixed data types more difficult.

Putting it all together

As previously mentioned, it is quite common for all of the above (and more) to be grouped together when talking about vectorization in numpy. However, as vectorization is also used in other contexts to describe more specific operations, this can be quite confusing.

Hopefully, it is all a little bit clearer as to what we are dealing with, and now we can move on to the practical side.

How much difference does numpy's implementation of vectorization really make?

A practical example

To demonstrate the effectiveness of vectorization in numpy we will compare a few different commonly used methods to apply mathematical functions, and also logic, using the pandas library.

pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool,
built on top of the Python programming language.

- pydata.org

Pandas is widely used when dealing with tabular data, and is also built on top of numpy, so I think it serves as a great medium for demonstrating the effectiveness of vectorization.

All the calculations that follow are available in a colab notebook

The data

The data will be a simple dataframe with two columns. Both columns will be comprised of 1 million rows of random numbers taken from a normal distribution.

df = pd.DataFrame({'series1':np.random.randn(1000000), 'series2':np.random.randn(1000000)})

which results in:

row	series1	series2
0	2.024360	-0.304465
1	-0.294511	-0.585608
2	-0.580776	-0.987834
3	1.403553	1.553986
4	-2.004211	-0.263476
...	...	...
999995	-0.448020	0.040024
999996	0.325896	0.574605
999997	1.679847	-1.103830
999998	0.568573	-1.695838
999999	-0.362537	1.556493

1000000 rows × 2 columns

The manipulation

Then the above dataframe will be manipulated by two different functions to create a third column 'series3'. This is a very common operation in pandas, for example, when creating new features for machine or deep learning:

Function 1 - a simple summation

def sum_nums(a, b):
	return a + b

Function 2 - logic and arithmetic

def categorise(a, b):
    if a < 0:
        return a * 2 + b
    elif b < 0:
        return a + 2 * b
    else:
        return None

Each of the above functions will be applied using different methods (some vectorized, some not) to see which performs the calculations over the 1 million rows the quickest.

The methods and the results

The processing methods that follow are arranged in order of speed. Slowest first.

Each method was run multiple times using the timeit library, and for both of the functions mentioned in the previous section. Once for the slower methods, up to 1000 times for the faster methods. This ensures the calculations don't run too long, and we get enough iterations to average out the run time per iteration.

The pandas apply method

The pandas apply method is very simple and intuitive. However, it is also one of the slowest ways for applying calculations on large datasets.

There is no optimisation of the calculation. It is basically performing a simple for loop. This method should be avoided unless the requirements of the function rule out all other methods.

# Function 1
series3 = df.apply(lambda df: sum_nums(df['series1'],df['series2']),axis=1)
# Function 2
series3 = df.apply(lambda df: categorise(df['series1'],df['series2']),axis=1)

Function	Iterations	Total Time (s)	Time per iteration (s)	Improvement in speed
1	1	11.60	11.60	Baseline
2	1	11.58	11.58	Baseline

Itertuples

Itertuples, in some simple implementations, is even slower than the apply method, but in this case it is used with list comprehension, so achieves almost a 20 times improvement in speed over the apply method.

Itertuples removes the overhead of dealing with a pandas Series and instead uses named tuples for the iteration^[1]. As previously mentioned, this particular implementation also benefits from the speed up list comprehension provides, by removing the overhead of appending to a list^[2].

Note: there is also a function called iterrows, but it is always slower, and therefore ignored for brevity.

# Function 1
series3 = [sum_nums(a, b) for a, b in df.itertuples(index=False)]
# Function 2
series3 = [categorise(a, b) for a, b in df.itertuples(index=False)]

Function	Iterations	Total Time (s)	Time per iteration (s)	Improvement in speed
1	10	6.12	0.612	x19
2	10	6.41	0.641	x18

List comprehension

The previous itertuples example also used list comprehension, but it seems this particular solution using 'zip' instead of itertuples is about twice as fast.

The main reason for this is the additional overhead introduced by the itertuples method. Itertuples actually uses zip internally, so any additional code to get to the point where zip is applied is just unnecessary overhead.

A great investigation into this can be found in this article. Incidentally, it also explains why iterrows is slower than itertuples.

# Function 1
series3 = [sum_nums(a, b) for a, b in zip(df['series1'],df['series2'])]
# Function 2
series3 = [categorise(a, b) for a, b in zip(df['series1'],df['series2'])]

Function	Iterations	Total Time (s)	Time per iteration (s)	Improvement in speed
1	100	29.31	0.293	x40
2	100	31.13	0.311	x37

Numpy vectorize method

This is a bit of an odd one. The method itself is called 'vectorize', but the truth is it is no where near as fast as the full on optimised vectorization that we will see in the methods that follow. Even numpy's own documentation states:

The vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop.

- numpy.org

However, it is true that the syntax used to implement this function is extremely simple and clear. On top of that, the method actually does a great job of speeding up the calculation, more so than any method we have tried up till now.

It is also more flexible than the methods that follow, and so is easier to implement in lots of situations without any messing about. Numpy vectorize is therefore a great method to use, and highly recommended.

It is just worth bearing in mind that although this method is quick, it is not even close to what is achievable with the fully optimised methods we are about to see, so it should not just be your go to method in all situations.

# Function 1
series3 = np.vectorize(sum_nums)(df['series1'],df['series2'])
# Function 2
series3 = np.vectorize(categorise)(df['series1'],df['series2'])

Function	Iterations	Total Time (s)	Time per iteration (s)	Improvement in speed
1	100	22.13	0.221	x52
2	100	21.41	0.214	x54

Pandas vectorization

Now we come to full on optimised vectorization.

The difference in speed is night and day compared to any method before, and a prime example of all the optimisations discussed in earlier sections of this article working together.

The pandas implementation is still an implementation of numpy under the hood, but the syntax is very, very straight forward. If you can express your desired calculation this way, you can't do much better in terms of speed without coming up with a significantly more complicated implementation.

Approximately, 7000 times faster than the apply method, and 130 times faster than the numpy vectorize method!

The downside, is that such simple syntax does not allow for more complicated logic statements to be processed.

# Function 1
series3 = df['series1'] + df['series2']
# Function 2
# N/A as a simple operation is not possible due to the included logic in the function.

Function	Iterations	Total Time (s)	Time per iteration (s)	Improvement in speed
1	1000	1.66	0.00166	x7000
-	-	-	-	-

Numpy vectorization

The final implementation is as close as we can get to implementing raw numpy whilst still having the inputs from a pandas dataframe. Even so, by stripping away any pandas overhead in the calculation, a 15% reduction in processing time is achieved when compared to the pandas implementation.

That is 8000 times faster than the apply method.

# Function 1
series3 = np.add(df['series1'].to_numpy(),df['series2'].to_numpy())
# Function 2
# N/A as a simple operation is not possible due to the included logic in the function.

Function	Iterations	Total Time (s)	Time per iteration (s)	Improvement in speed
1	1000	1.42	0.00142	x8000
-	-	-	-	-

Conclusion

I hope this article has helped to clarify some of the jargon that exists especially in relation to vectorization, and therefore allowed you to have a better understanding of which methods would be most appropriate depending on your particular situation.

As a general rule of thumb if you are dealing with large datasets of numerical data, vectorized methods in pandas and numpy are your friend:

If the calculation allows, try to use numpy's inbuilt mathematical functions^[3]
Pandas' mathematical operations are also a good choice
If you require more complicated logic, use numpy's vectorize^[4] method
Failing all of the above, it is a case of deciding exactly what functionality you need, and choosing one of the slower methods as appropriate (list comprehension, intertuples, apply)

If you find yourself in a situation where you need both speed and more flexibility, then you are in a particularly niche situation. You may need to start looking into implementing your own parallelisation, or writing your own bespoke numpy functions^[5]. All of which is possible.

References

[1] https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.itertuples.html

[2] Mazdak, Why is a list comprehension so much faster than appending to a list? (2015), stackoverflow.com

[3] https://numpy.org/doc/stable/reference/routines.math.html

[4] https://numpy.org/doc/stable/reference/generated/numpy.vectorize.html

[5] https://numpy.org/doc/stable/user/c-info.ufunc-tutorial.html

How to Pick the Best Graphics Card for Machine Learning

Mon, 19 Sep 2022 01:00:00 GMT

When dealing with machine learning, and especially when dealing with deep learning and neural networks, it is preferable to use a graphics card to handle the processing, rather than the CPU. Even a very basic GPU is going to outperform a CPU when it comes to neural networks.

But which GPU should you buy? There is a lot of choice, and it can get confusing and expensive very quickly. I will therefore try to guide you on the relevant factors to consider, so that you can make an informed choice based on your budget and particular modelling requirements.

Why is a GPU preferable over a CPU for Machine Learning?

A CPU (Central Processing Unit) is the workhorse of your computer, and importantly is very flexible. It can deal with instructions from a wide range of programs and hardware, and it can process them very quickly. To excel in this multitasking environment a CPU has a small number of flexible and fast processing units (also called cores).

A GPU (Graphics Processing Unit) is a little bit more specialised, and not as flexible when it comes to multitasking. It is designed to perform lots of complex mathematical calculations in parallel, which increases throughput. This is achieved by having a higher number of simpler cores, sometimes thousands, so that many calculations can be processed all at once.

Image by Ahmed Gad from Pixabay

This requirement of multiple calculations being carried out in parallel is a perfect fit for:

graphics rendering - moving graphical objects need their trajectories calculated constantly, and this requires a large amount of constant repeat parallel mathematical calculations.
machine and deep learning - large amounts of matrix/tensor calculations, which with a GPU can be processed in parallel.
any type of mathematical calculation that can be split to run in parallel.

I think the best summary I have seen is on Nvidia's own blog:

CPU	GPU
Central Processing Unit	Graphics Processing Unit
Several cores	Many cores
Low latency	High throughput
Good for serial processing	Good for parallel processing
Can do a handful of operations at once	Can do thousands of operations at once

Source

Tensor Processing Unit (TPU)

With the boom in AI and machine/deep learning there are now even more specialised processing cores called Tensor cores. These are faster and more efficient when performing tensor/matrix calculations. Exactly what you need for the type of mathematics involved in machine/deep learning.

Although there are dedicated TPUs, some of the latest GPUs also include a number of Tensor cores, as you will see later in this article.

Nvidia vs AMD

This is going to be quite a short section, as the answer to this question is definitely: Nvidia

You can use AMD GPUs for machine/deep learning, but at the time of writing Nvidia's GPUs have much higher compatibility, and are just generally better integrated into tools like TensorFlow and PyTorch.

I know from my own experience that trying to use an AMD GPU with TensorFlow requires using additional tools (ROCm), which tend to be a bit fiddly, and sometimes leave you with a not quite up to date version of TensorFlow/PyTorch, just so you can get the card working.

This situation may improve in the future, but if you want a hassle free experience, it is better to stick with Nvidia for now.

GPU Features

Picking out a GPU that will fit your budget, and is also capable of completing the machine learning tasks you want, basically comes down to a balance of four main factors:

How much RAM does the GPU have?
How many CUDA and/or Tensor cores does the GPU have?
What chip architecture does the card use?
What are your power draw requirements (if any)?

The subsequent sub sections will look into each of these areas, and hopefully give you a better grasp of what matters to you.

GPU RAM

The answer to this is the more the better! Very helpful I know...

It really comes down to what you are modelling, and how big those models are. For example if you are dealing with images, videos or audio, then by definition you are going to be dealing with quite a large amount of data, and GPU RAM will be an extremely important consideration.

There are always ways to get around running out of memory (e.g. reducing the batch size). However, you want to limit the amount of time you have to spend messing about with code just to get around memory requirements, so a good balance for your requirements is essential.

Image by OpenClipart-Vectors from Pixabay

As a general rule of thumb I would suggest the following:

4GB - The absolute minimum I would consider, this will work well in most cases as long as you are not dealing with overly complicated models, or large amounts of images, videos or audio. Great if you are just starting out and want to experiment without breaking the bank. The improvements over a CPU will still be night and day.

8GB - I would say this is a good middle ground. You can get most tasks done without hitting RAM limits, but you will have problems with more complicated models with images, video or audio.

12GB - This I would describe as optimal without being ridiculous. You can deal with most larger models, even those that deal with images, video or audio.

12GB+ - The more the better, you will be able to handle larger datasets and bigger batch sizes. However, beyond 12GB is where the prices really start to ramp up.

In my experience I would say it is better, on average, to opt for a card that is 'slower' with more RAM, if the cost is the same. Remember, the advantage of a GPU is high throughput, and this is heavily dependent on available RAM to feed the data through the GPU.

CUDA Cores and Tensor Cores

This is fairly simple really. The more CUDA (Compute Unified Device Architecture) cores / Tensor cores the better.

The other items such as RAM and chip architecture (see the next section) should probably be considered first, and then look at cards with the highest number of CUDA/tensor cores from your narrowed down selection.

For machine/deep learning Tensor cores are better (faster and more efficient) than CUDA cores. This is due to them being designed precisely for the calculations that are required in the machine/deep learning domain.

The reality is it doesn't matter a great deal, CUDA cores are plenty fast enough. If you can get a card which includes tensor cores too, that is a good plus point to have, just don't get too hung up on it.

Moving forward you will see "CUDA" mentioned a lot, and it can get confusing, so to summarise:

CUDA cores - these are the physical processors on the graphics cards, typically in their thousands.
CUDA 11 - The number may change, but this is referring to the software/drivers that are installed to allow the graphics card to work. New releases are made regularly, and it can be installed like any other software.
CUDA generation (or compute capability) - this describes the capability of the graphics card in terms of it's generational features. This is fixed in hardware, and so can only be changed by upgrading to a new card. It is distinguished by numbers and a code name. Examples: 3.x [Kepler], 5.x [Maxwell], 6.x [Pascal], 7.x [Turing] and 8.x [Ampere].

Chip architecture

This is actually more important than you might think. As I mentioned earlier we are basically discarding AMD at this point, so in terms of generations of chip architecture we only have Nvidia.

Photo by Manuel on Unsplash

The main thing to look out for is the "Compute Capability" of the chipset, sometimes called "CUDA generation". This is fixed for each card, so once you buy the card you are stuck with whatever compute capability the card has.

It is important to know what the compute capability of the card is for two main reasons:

significant feature improvements
deprecation

Significant feature improvement

Let's start with a significant feature improvement. Mixed precision training:

There are numerous benefits to using numerical formats with lower precision than 32-bit floating point. First, they require less memory, enabling the training and deployment of larger neural networks. Second, they require less memory bandwidth which speeds up data transfer operations. Third, math operations run much faster in reduced precision, especially on GPUs with Tensor Core support for that precision. Mixed precision training achieves all these benefits while ensuring that no task-specific accuracy is lost compared to full precision training. It does so by identifying the steps that require full precision and using 32-bit floating point for only those steps while using 16-bit floating point everywhere else.

-Nvidia Learning Performance Documentation

It is only possible to use mixed precision training if you have a GPU with compute capability 7.x (Turing) or higher. That is basically the RTX 20 series or newer, or the RTX, "T" or "A" series on desktop/server.

The main reason mixed precision training is such an advantage when considering a new graphics card is that it lowers RAM usage, so by having a slightly newer card your RAM requirements are reduced.

Deprecation

Then we move to the other end of the scale.

If you have particularly high RAM requirements, but not enough money for a high end card, then it may be the case that you will opt for an older model of GPU on the second hand market.

However, there is quite a large downside...the card is end of life.

A prime example of this is the Tesla K80, which has 4992 CUDA cores and 24GB of RAM. It originally retailed at about USD 7000.00 back in 2014. I just had a look on e-bay in the UK, and it is going for anything from GBP 130.00 (USD/EUR 150) to GBP 170.00 (USD/EUR 195)! That is a lot of RAM for such a small price.

However, there is quite a large downside. The K80 has a compute capability of 3.7 (Kepler), which is deprecated from CUDA 11 onwards (the current CUDA version is 11). That means the card is end of life, and won't work on future releases of CUDA drivers. A great pity really, but something to bear in mind, as it is very tempting.

Graphics Cards vs Workstation/Server Cards

Nvidia basically splits their cards into two sections. There are the consumer graphics cards, and then cards aimed at desktops/servers (i.e. professional cards).

There are obviously differences between the two sections, but the main thing to bear in mind is that the consumer graphics cards will generally be cheaper for the same specs (RAM, CUDA cores, architecture). However, the professional cards will generally have better build quality, and lower energy consumption.

Photo by Elias Gamez on Pexels

Looking at the higher end (and very expensive) professional cards you will also notice that they have a lot of RAM (the RTX A6000 has 48GB for example, and the A100 has 80GB!). This is due to the fact that they are typically aimed directly at 3D modelling, rendering, and machine/deep learning professional markets, which require high levels of RAM. Again, if you have those sort of requirements you will likely not need advice on what to purchase!

In summary, you will likely be best sticking to the consumer graphics market, as you will get a better deal.

Recommendation

Finally, I thought I would actually make some recommendations based on budget and requirements. I have split this into three sections:

Low budget
Medium budget
High budget

Please bear in mind the high budget does not consider anything beyond top end consumer graphics cards. If you really have a very high budget you should be looking at the professional card series, such as the Nvidia A series of cards which can run costs up into many thousands.

I have included one card that is only available on the second hand market in the low budget section. This is mainly because I think in the low budget section it is worth considering second hand cards.

Photo by Nana Dua on Unsplash

I have also included professional desktop series cards in the mix as well (T600, A2000 and A4000). You will notice that some of the specs are a little worse than comparable consumer graphics cards, but power draw is significantly better, which may be of concern for some people.

Low Budget (Less than GBP 220 - EUR/USD 250)

GPU	RAM	Compute Capability	CUDA Cores	Mixed Precision	Power Consumption
GTX 1070 / 1070ti / 1080	8GB	6.1 (Pascal)	1920 / 2432 / 2560	No	150W / 180W / 180W
GTX 1660	6GB	7.5 (Turing)	1408	Yes	120W
T600	4GB	7.5 (Turing)	640	Yes	40W

Medium Budget (Less than GBP 440 - EUR/USD 500)

GPU	RAM	Compute Capability	CUDA Cores	Tensor Cores	Mixed Precision	Power Consumption
RTX 2060 12GB	12GB	7.5 (Turing)	2176	272 (Gen 2)	Yes	184W
RTX A2000 6GB	6GB	8.6 (Ampere)	3328	104 (Gen 3)	Yes	70W
RTX 3060 12GB	12GB	8.6 (Ampere)	3584	112 (Gen 3)	Yes	170W

High Budget (Less than GBP 1050 - EUR/USD 1200)

GPU	RAM	Compute Capability	CUDA Cores	Tensor Cores	Mixed Precision	Power Consumption
RTX 3080 Ti	12GB	8.6 (Ampere)	10240	320 (Gen 3)	Yes	350W
A4000 16GB	16GB	8.6 (Ampere)	6144	192 (Gen 3)	Yes	140W
RTX 3090	24GB	8.6 (Ampere)	10496	328 (Gen 3)	Yes	350W

Other Options

If you have decided the outlay and bother of getting a graphics card is not for you, you can always take advantage of Google Colab, which gives you access to a GPU for free. Just bear in mind there are time limitations to this, and the GPU is automatically assigned, so having your own graphics card is usually worth it in the long run.

At the time of writing the following GPUs are available through Colab:

GPU	RAM	Compute Capability	CUDA Cores	Tensor Cores
Tesla K80	12GB	3.7	2496	0
Tesla P100	16GB	6.0	3584	0
Tesla T4	16GB	7.5	2560	320

Note: I mentioned earlier in the article that the K80 has 24GB of RAM and 4992 CUDA cores, and it does. However, the K80 is an unusual beast in that it is basically two K40 cards bolted together. This means that when you use a K80 in Colab, you are actually given access to half the card, so only 12GB and 2496 CUDA cores.

Conclusion

There is a lot of choice out there, and it can be very confusing. Hopefully you have come away from this article with a much better idea of what will fit your particular requirements.

Is there a particular graphics card that you think deserves a special mention? Let me know in the comments.

Mixed Precision Training - Less RAM, More Speed

Sat, 24 Sep 2022 01:00:00 GMT

When it comes to large complicated models it is essential to reduce the model training time as much as possible, and utilise the available hardware efficiently. Even small gains per batch or epoch are very important.

Mixed precision training can both significantly reduce GPU RAM utilisation, as well as speeding up the training process itself, all without any loss of precision in the outcome.

This article will show (with code examples) the sort of gains that can actually be attained, whilst also going over the requirements to use mixed precision training in your own models.

Introduction

The first half of this article is aimed at giving an overview of what mixed precision is, and when, why and how to use it.

The second half goes through the results of a comparison between 'normal' and mixed precision training on a set of dummy images. The images are trained through a multi-layer Conv2D neural network in TensorFlow, and both RAM usage and execution speed are monitored throughout.

All the code relevant to the comparison is available in a colab notebook

What exactly is mixed precision?

Before we dive into what mixed precision is, it is probably a good idea to outline what we are referring to when we say 'precision' in this particular context.

Precision in this case is basically referring to how a floating point number is stored i.e. how much space it takes up in memory. The smaller the memory footprint, the less accurate the number. There are basically three options:

Half precision - 16-bit (float16) - low level of storage used to represent number, low level of accuracy
Single precision - 32-bit (float32) - medium level of storage used to represent number, medium level of accuracy
Double precision - 64-bit (float64) - high level of storage used to represent number, high level of accuracy

Typically with machine learning / deep learning and neural networks, you will be dealing with single precision 32-bit floating point numbers.

Image by InspiredImages from Pixabay

However, in almost all cases it is possible for calculations to be run using 16-bit floating point numbers instead of 32-bit floating point numbers, without any degradation of the accuracy of the model.

Mixing precisions

The ideal, and simplest solution, is to use a mixture of 16-bit and 32-bit floating point numbers. Calculations can be run as fast as possible using lower precision 16-bit floating point numbers, and then the inputs and outputs can be stored as 32-bit floating point variables to ensure a high level of accuracy is preserved and there are no compatibility issues on the output.

This combination is what is referred to as 'Mixed Precision'.

Why should I use mixed precision?

There are two main reasons:

There will be a significant improvement in GPU RAM usage. The difference can be as much as 50% less GPU RAM utilisation
There can be a significant speed up in time taken to run through the model

Using mixed precision in TensorFlow could:

improve performance by more than 3 times on modern GPUs and 60% on TPUs

-tensorflow.org

The RAM usage reduction alone is a big deal. This will allow larger batch sizes to be utilised, or open the door to larger and more intensive models being possible on the same hardware.

We will of course see actual results for these two factors in the comparison later in the article.

What are the requirements to use mixed precision?

For mixed precision training to be an advantage you will need one of the following:

A Nvidia GPU with compute compatibility of 7.0 or above (you can get more details on 'compute compatibility' and why Nvidia specifically in my previous article here.)
A TPU (Tensor Processing Unit)

Photo by Nana Dua on Unsplash

Although you can use other GPUs for mixed precision, and it will run. You won't gain any real speed improvements without the items detailed above. However, if you are only looking for gains in RAM usage then it may still be worth it.

Older GPUs offer no math performance benefit for using mixed precision, however memory and bandwidth savings can enable some speedups.

-tensorflow.org

When should I use mixed precision?

The simple answer to this question is almost all the time, as the advantages greatly outweigh the disadvantages in most cases.

The only thing to note is that if your models are relatively uncomplicated and small, you will likely not realise the difference. The larger and more complicated the models get, the more significant an advantage mixed precision is.

How do I use mixed precision?

In TensorFlow it is extremely easy, I'm not that familiar with PyTorch, but I can't imagine it would be particularly difficult to implement either.

from tensorflow.keras import mixed_precision
mixed_precision.set_global_policy('mixed_float16')

...and that's it.

The only caveat to the above is that you should ensure that the inputs and outputs of the model are always float32. The inputs will likely be in float32 anyway, but just to be sure you can implicitly apply the dtype. For example:

images = tf.random.uniform(input_shape, minval=0.0, maxval=1.0, seed=SEED, dtype=tf.float32)

To ensure your output from your model is in float32, you can separate out the activation of the last layer of your model. For example:

# Simple layer stack using the funcitonal API with separated activation layer as output

layer1 = tf.keras.layers.Conv2D(128,2)(inputs)
layer2 = tf.keras.layers.Conv2D(128,1)(layer1)
layer3 = tf.keras.layers.Conv2D(128,1)(layer2)
layer4 = tf.keras.layers.Flatten()(layer3)
layer5 = tf.keras.layers.Dense(1)(layer4)
output_layer = tf.keras.layers.Activation('sigmoid', dtype=tf.float32)(layer5)

Custom training loops

Applying mixed precision to your models really is a simple as described in the previous section.

However, if you are in a situation where you are not using 'model.fit' because you are implementing your own training loop, then there are a few more steps to be aware of as you have to manually deal with loss scaling.

If you use tf.keras.Model.fit, loss scaling is done for you so you do not have to do any extra work. If you use a custom training loop, you must explicitly use the special optimizer wrapper tf.keras.mixed_precision.LossScaleOptimizer in order to use loss scaling.

-tensorflow.org

This is important as float16 values are prone to 'underflow' and 'overflow' due to the smaller storage available compared to float32. All this essential means is that:

values above 65504 will overflow to infinity and values below 6.0×10−8 will underflow to zero.

-tensorflow.org

To avoid this a strategy called loss scaling is utilised to mitigate this problem. For a deeper understanding I suggest taking a look at the mixed precision guide on tensorflow.org.

TPUs

If you are lucky enough to have access to a dedicated TPU (Tensor Processing Unit) then it is just worth noting that you should be using data type ''bfloat16" rather than "float16".

It is no harder to implement, and doesn't suffer from the loss scaling problem as mentioned in the previous section.

from tensorflow.keras import mixed_precision
mixed_precision.set_global_policy('mixed_bfloat16')

A practical example

As an example of the potential gains, I have made available a colab notebook so that you can see the benefits for yourself. There are some notes at the beginning of the notebook in relation to the GPU you must use, so please make sure you read those to get the most out of the notebook.

I will go through the outcomes from this notebook in the following subsections.

The data

The data is random uniform noise formatted into the shape of a batch of images.

# create dummy images based on random data
SEED = 12
tf.random.set_seed(SEED);
total_images = 800
input_shape = (total_images, 256, 256, 3) # (batch, height, width, channels)
images = tf.random.uniform(input_shape, minval=0.0, maxval=1.0, seed=SEED, dtype=tf.float32)

It is important to note that I have explicitly set the data type to be float32. In this case it would have made no difference as this is the default for the function. However, this may not always be the case depending on where your data comes from.

An example image looks as follows:

Image by author

I also created random binary labels so that the model can be a binary classification model.

labels = np.random.choice([0, 1], size=(total_images,), p=[0.5,0.5])

The model

The model has been chosen to be simple, but complicated enough to use a reasonable amount of RAM, and have a decent batch run time. This ensures that any differences between the mixed precision and 'normal' run are distinguishable. These are the layers of the model:

layer1 = tf.keras.layers.Conv2D(128,2)
layer2 = tf.keras.layers.Conv2D(128,1)
layer3 = tf.keras.layers.Conv2D(128,1)
layer4 = tf.keras.layers.Flatten()
layer5 = tf.keras.layers.Dense(1)
output_layer = tf.keras.layers.Activation('sigmoid',dtype=tf.float32)

Again, take note that the output activation layer is cast to float32. This makes no difference on the 'normal' run, but is essential for the mixed precision run.

The test

The model mentioned in the previous section was run using the following parameters:

total images = 800
image size = 256 x 256
batch size = 50
epochs = 10

Overall run time and epoch runtime

The images are then run through once using the timeit module to get an overall run time.

The epoch run times are also printed.

GPU RAM usage

To get GPU RAM usage information the following function is used:

tf.config.experimental.get_memory_info('GPU:0')

This outputs the current and peak GPU RAM usage. Before each run the peak usage is reset and compared to the current GPU RAM usage (they should therefore be the same). Then at the end of the run the same comparison is made. This allows the calculation of the actual GPU RAM used during the run.

The Results

Single precision (float32) model:

Epoch 1/10
16/16 [==============================] - 10s 463ms/step - loss: 90.4716 - accuracy: 0.5038
Epoch 2/10
16/16 [==============================] - 8s 475ms/step - loss: 9.1019 - accuracy: 0.6625
Epoch 3/10
16/16 [==============================] - 8s 477ms/step - loss: 1.6142 - accuracy: 0.8737
Epoch 4/10
16/16 [==============================] - 8s 475ms/step - loss: 0.2461 - accuracy: 0.9488
Epoch 5/10
16/16 [==============================] - 8s 482ms/step - loss: 0.0486 - accuracy: 0.9800
Epoch 6/10
16/16 [==============================] - 8s 489ms/step - loss: 0.0044 - accuracy: 0.9975
Epoch 7/10
16/16 [==============================] - 8s 494ms/step - loss: 7.3721e-05 - accuracy: 1.0000
Epoch 8/10
16/16 [==============================] - 8s 497ms/step - loss: 1.4208e-05 - accuracy: 1.0000
Epoch 9/10
16/16 [==============================] - 8s 496ms/step - loss: 1.2936e-05 - accuracy: 1.0000
Epoch 10/10
16/16 [==============================] - 8s 490ms/step - loss: 1.1361e-05 - accuracy: 1.0000

RAM INFO:

Current: 0.63 GB, Peak: 9.18 GB, USED MEMORY FOR RUN: 8.55 GB

TIME TO COMPLETE RUN: 79.73

Mixed precision (mixed_float16) model:

Epoch 1/10
16/16 [==============================] - 15s 186ms/step - loss: 71.8095 - accuracy: 0.5025
Epoch 2/10
16/16 [==============================] - 3s 184ms/step - loss: 15.2121 - accuracy: 0.6000
Epoch 3/10
16/16 [==============================] - 3s 182ms/step - loss: 4.4640 - accuracy: 0.7900
Epoch 4/10
16/16 [==============================] - 3s 183ms/step - loss: 1.1157 - accuracy: 0.9187
Epoch 5/10
16/16 [==============================] - 3s 183ms/step - loss: 0.2525 - accuracy: 0.9600
Epoch 6/10
16/16 [==============================] - 3s 181ms/step - loss: 0.0284 - accuracy: 0.9925
Epoch 7/10
16/16 [==============================] - 3s 182ms/step - loss: 0.0043 - accuracy: 0.9962
Epoch 8/10
16/16 [==============================] - 3s 182ms/step - loss: 7.3278e-06 - accuracy: 1.0000
Epoch 9/10
16/16 [==============================] - 3s 182ms/step - loss: 2.4797e-06 - accuracy: 1.0000
Epoch 10/10
16/16 [==============================] - 3s 182ms/step - loss: 2.5154e-06 - accuracy: 1.0000

RAM INFO:

Current: 0.63 GB, Peak: 4.19 GB, USED MEMORY FOR RUN: 3.57 GB

TIME TO COMPLETE RUN: 42.16

I think that is fairly conclusive:

Data Type	Epoch run time [s]	Overall run time [s]	GPU RAM Usage [GB]
float32	8	79.73	8.55
mixed_float16	3	42.16	3.57
Improvement	Almost 3x faster	Almost 2x faster	Less than half the RAM usage

One thing you may note in the above results is that the initial epoch for the mixed precision run takes five times longer than the subsequent epochs, and even longer than the float32 run. This is normal, and is due to the optimisations that TensorFlow runs at the start of the learning process. However, even with this initial deficit, it doesn't take long for the mixed precision model to catch up and surpass the float32 model.

The longer initial epoch for mixed precision also helps to illustrate why smaller models may not see the benefits, as there are initial overheads that need to be overcome to realise the advantages of mixed precision.

This also happens to serve as a great example of overfitting. Both methods managed to achieve 100% accuracy on completely random data with completely random labels!

The future

The trend for lower precision calculations seems to be gaining traction, as with the latest generations of GPUs from Nvidia there are now implementations such as TensorFloat-32, which:

automatically uses lower precision math in certain float32 ops such as tf.linalg.matmul.

-tensorflow.org

It is also the case that:

TPUs do certain ops in bfloat16 under the hood even with the default dtype policy of float32

-tensorflow.org

As such, as time goes on it may not be necessary to actually implement mixed precision directly as it will all be taken care of under the hood.

However, We are not there yet, so for now it is still worth the effort to consider utilising mixed precision training.

Conclusion

The only conclusion to draw is that mixed precision is an excellent tool to speed up training, but more importantly free up GPU RAM.

Hopefully, this article has helped you get a grasp of what mixed precision is all about, and I would encourage you to have a play around with the colab notebook to see if it fits your particular requirements, and get a feel for the benefits it could bring.

The Best Methods for One-Hot Encoding Your Data

Sun, 30 Oct 2022 01:00:00 GMT

Pre-processing data before feeding it into a machine / deep learning model is one of the most important phases in the whole process. Without properly pre-processed data it won’t matter how advanced and slick your model is, it will ultimately be inefficient and inaccurate.

One Hot Encoding is probably the most commonly utilised pre-processing method for independent categorical data, ensuring that the model can interpret the input data fairly, and without bias.

This article will explore the three most common methods of encoding categorical data using the one-hot method, and discuss why you would want to use this technique in the first place.

Introduction

The following methods are going to be compared and discussed in this article:

Pandas — get_dummies()
Scikit-Learn — OneHotEncoder()
Keras — to_categorical()

All methods basically achieve the same outcome. However, they go about it in completely different ways, and have different features and options.

So which of these methods would be best suited for your specific circumstances?

What is One-Hot Encoding?

Before we dive in I though it might be worthwhile giving a quick primer as to why you might want to use this method in the first place.

One hot encoding is basically a way of preparing categorical data to ensure the categories are viewed as independent of each other by the machine learning / deep learning model.

A solid example

Let’s give a practical example to really ram the idea home. We have three categories:

Chicken
Rock
Gun

They are in no way related to each other, completely independent.

To feed these categories into a machine learning model we need to turn them into numerical values, as machine / deep learning models cannot deal with any other type of input. So how best to do this?

A chicken on the run. Photo by James Wainscoat on Unsplash

The most simple way is just to assign each category a number:

Chicken
Rock
Gun

The problem with this approach (called ordinal encoding) is that the model can infer a relationship between the categories as the numbers follow each other.

Is a gun more important than a chicken because it has a higher number? Is a chicken half a rock? If you have three chickens, is that the same as a gun?

What if these values are labels, if the model outputs 1.5 as the answer, is that some sort of chicken-rock? All of these statements are nonsense, but as the model only sees numbers, not the names that we see, inferring these things is perfectly feasible for the model.

To avoid this we need complete separation of the categories. This is what One-Hot Encoding achieves:

Chicken	Rock	Gun
1	0	0
0	1	0
0	0	1

The values are only ever one or zero (on or off). One means it is that thing, and zero means it isn’t.

So on the first row you have a chicken (no rock and no gun), the second row has a rock (no chicken and no gun) etc. As the values are either on or off, there is no possibility of relating one to the other in any way.

A quick note

Before I dive into each of the three methods in more detail I just wanted to point out that I will be using an alternative to Colab, which I would typically use to make code available for the article.

The alternative I will be using is deepnote. It is essentially the same as Colab in that it lets you run Jupyter notebooks in an online environment (there are some differences, which I won’t go into here, but check out the website to learn more).

The main reason for this is that to demonstrate the latest features for some of the methods in this article, I needed access to Pandas 1.5.0 (the latest release at the time of writing), and I can’t seem to achieve this in Colab.

However, in deepnote I can specify a Python version (in this case 3.10), and also make my own requirements.txt to ensure the environment installs Pandas 1.5.0, not the default version.

It also allows very simple live embeds directly from the Jupyter notebook into this article (as you will see), which is very useful.

I will still make the notebook available in colab as usual, but some of the code won’t run, so just bear that in mind.

The data

The data¹²³ is basically related to the effects of alcohol consumption on exam results. Not something that you need to remember, but incase you are interested…

Photo by Vinicius “amnx” Amano on Unsplash

As ever I have made the data available in a Jupyter notebook. You can access this in either deepnote:

(as previously mentioned, Pandas 1.5.0 is available in deepnote. Just activate Python 3.10 in the "Environment" section on the right, and create a text file in the "Files" section on the right called "requirements.txt" with the line "pandas==1.5.0" in it. Then run the notebook.)

or colab:

(some methods that follow will not work as Pandas 1.5.0 is required)

I have selected a dataset that includes a wide range of different categorical and non-categorical columns so that it is easy to see how each of the methods works depending on the datatype. The columns are as follows:

sex — binary string (‘M’ for male and ‘F’ for female)
age — standard numerical column (int)
Medu — Mother’s education — Multiclass integer representation (0 [none], 1 [primary education], 2 [5th to 9th grade], 3 [secondary education] or 4 [higher education])
Mjob — Mother’s job — Multiclass string representation — (‘teacher’, ‘health’ care related, civil ‘services’ (e.g. administrative or police), ‘at_home’ or ‘other’)
Dalc — Workday alcohol consumption — Multiclass graduated integer representation (from 1 [very low] to 5 [very high])
Walc — Weekend alcohol consumption — Multiclass graduated integer representation (from 1 [very low] to 5 [very high])
G3 — Final grade (the label) — Multiclass graduated integer representation (numeric: from 0 to 20)

An example of the top five rows is as follows:

and the data types:

Pandas — get_dummies()

Photo by Pascal Müller on Unsplash

pandas.get_dummies(data, prefix=None, prefix_sep=’_’, dummy_na=False, columns=None, sparse=False, drop_first=False, dtype=None)

Documentation

I would describe the get_dummies() method from Pandas as a very middle of the road one-hot encoder. It keeps things simple while giving a reasonable amount of options to allow you to adjust to the most common use cases.

You can very simply just pass a Pandas dataframe to get_dummies() and it will work out which columns are most suitable for one hot encoding.

However, this is not the best way to approach things as you will see:

If you review the output above you will see that only columns with type ‘Object’ have been one-hot encoded (sex and MJob). Any integer datatype columns have been ignored, which in our case is not ideal.

However, you can specify the columns you wish to encode as follows:

One thing to note is that when using get_dummies() it keeps everything within the dataframe. There are no extra arrays to deal with. It is just all neatly kept in one place. This is not the case with OneHotEncoder() or to_categorical() methods, which will be discussed in subsequent sections.

There may be specific circumstances where it is advisable, or useful to drop the first column of each one hot encoded series (for example to avoid multi-colinearity). get_dummies() has this ability built in:

Note in the above how, for example, “Medu_0” is now missing.

The way this now works is that if Medu_1 to Medu_4 are all zero, this effectively means Medu_0 (the only other alternative) is “selected”.

Previously, when Medu_0 was included (i.e. drop_first wasn’t used), there would never have been a case where all values were zero. So in effect by dropping the column we don’t lose any information about the categories, but we do reduce the overall amount of columns, and therefore the processing power needed to run the model.

There are more nuanced things to consider when deciding if dropping a column is appropriate, but as that discussion would warrant a whole article of it’s own, I will leave it for you to look into.

Additional options

Apart from ‘drop_first’ there are also additional methods such as ‘sparse’ to produce a sparse matrix, and ‘dummy_na’ to help deal with NaN values that may be in your data.

There are also a couple of customisations available for the prefix and separators, should you need that level of flexibility.

Reversing get_dummies() with from_dummies()

There was until very recently no available method for reversing get_dummies() from the Pandas library. You would have had to do this manually.

However, as of Pandas 1.5.0 there is a new method called from_dummies():

pandas.from_dummies(data, sep=None, default_category=None)

Documentation

This allows the reversal to be achieved without writing your own method. It can even handle the reversal of a one-hot encoding that utilised ‘drop_first’ with the use of the ‘default_category’ parameter as you will see below:

To reverse when you have used ‘drop_first’ in the encoding you must specify the dropped items:

scikit-learn — OneHotEncoder()

Photo by Kelly Sikkema on Unsplash

The OneHotEncoder() method from Scikit-Learn is probably the most comprehensive of all the available methods for one hot encoding.

sklearn.preprocessing.OneHotEncoder(*, categories='auto', drop=None, sparse=True, dtype=<class 'numpy.float64'>, handle_unknown='error', min_frequency=None, max_categories=None)

Documentation

As you can see the method inputs above can handle:

automatically picking out the categories for one hot encoding
drop columns (not just the first, there are more extensive options available)
produce sparse matrices
handle categories that may appear in future datasets (handle_unknown)
you can limit the amount of categories returned from the encoding based on frequency or a maximum number of categories

The method also uses the fit-transform methodology, which is very useful for using this method in your input pipelines for machine and deep learning.

The encoder

One of the differences between this method and all the others is that you create an encoder ‘object’, which stores all the parameters that will be used to encode the data.

This can therefore be referred back to, re-used and adjusted at later points in your code, making it a very flexible approach.

Once the encoder has been instantiated we can one-hot encode some data:

In this case I have used the ‘fit_transform’ method, but as with all sklearn methods that follow the ‘fit’/’transform’ pattern you can also fit and transform the data in separate steps.

What OneHotEncoder does is it extracts the columns that it thinks should be one-hot encoded and returns them as a new array.

This is different to get_dummies(), which keeps the output in the same dataframe. If you want to keep all your data contained within a dataframe with minimal effort then this is something worth considering.

It should also be noted that OneHotEncoder recognises more input columns that get_dummies() when on ‘auto’ as you can see below.

Regardless, it is still good practise to specify the columns you wish to target.

For consistency moving forward, I will encode the same columns as we looked at previously with get_dummies():

Columns that have been encoded, and the parameters of the encoder:

Reversing OneHotEncoder

There is a very simple method for reversing the encoding, and as the encoder is saved as it’s own object (in this case ‘skencoder’), then all the original parameters used to do the one-hot encoding are saved within this object. This makes reversal very easy:

Other useful information

Another advantage to using OneHotEncoder is that there are a wealth of attributes and helper methods that give you access to the information used in the encoding. I have provided some examples below:

Attributes

Methods

Advanced Features

As mentioned earlier OneHotEncoder has quite a lot of useful features making it a very flexible method to use.

I will touch on some of these methods below.

Min frequency

This can be used to limit the encoded categories. If you have a feature that is dominated by a few significant categories, but has a lot of smaller categories, then you can effectively group the smaller categories into a single ‘other’ category.

You may find that you don’t want to specify an exact amount of records for the infrequent categories. In this case you could specify a minimum amount of records compared to the overall amount of records available. To do this you specify a fraction of the total count.

In our case there are 395 records, so to achieve the same outcome as specifying exactly 60 records as the limit, we could specify 60 / 395 = 0.152, or for simplicity 0.16 (which basically means that a category has to have 16% of the total count to be counted as significant).

Max categories

Another way to approach the problem is to specify a maximum number of categories.

Handle Unknown

Handle unknown is an extremely useful feature, especially when used in a pipeline for machine learning, or neural network model.

It essentially allows you to plan for a case in the future where another category may appear, without breaking your input pipeline.

For example, you may have a feature such as ‘Medu’, and in the future for some reason a ‘PhD’ category above the final category of ‘higher education’ is added to the input data. In theory, this additional category would break your input pipeline, as the amount of categories has changed.

Handle unknown allows us to avoid this.

Although I’m not going to give a concrete example of this, it is very easy to understand, especially if you have read the previous two sections on ‘max_categories’ and ‘min_frequency’.

Setting options:

‘error’ : this will just raise an error if you try to add additional category, you could say this is standard behaviour
‘ignore’ : this will cause any extra categories to be encoded with all zeros, so if there were originally 3 categories [1,0,0], [0,1,0] and [0,0,1] then the additional category (or categories) will be encoded as [0,0,0]. When inverted this will have the value ‘None’.
‘infrequent_if_exist’ : if you have implemented ‘max_categories’ or ‘min_frequency’ in your encoder, then the additional category will be mapped to ‘xxx_infrequent_sklearn’ along with any infrequent categories. Otherwise it will be treated exactly the same as ‘ignore’.

Important note: you cannot use handle_unknown=’ignore’ AND the drop category parameter (e.g. drop: ‘first’) at the same time. This is because they both produce a category with all zeros, and therefore conflict.

Drop

As with Pandas from_dummies() you have the option to drop categories, although the options are a little more extensive.

Here are the options (as per the documentation):

None : retain all features (the default).
‘first’ : drop the first category in each feature. If only one category is present, the feature will be dropped entirely.
‘if_binary’ : drop the first category in each feature with two categories. Features with 1, or more than 2 categories, are left intact.
array : drop[i] is the category in feature X[:, i] that should be dropped.

‘first’

The first entry of each category will be dropped (‘sex_F’, ‘Medu_0’ and ‘Mjob_at_home’).

‘if_binary’

Only features with exactly two categories will be affected (in our case only ‘sex_F’ is dropped).

‘array’

In this case you can pick exactly which category from each feature should be dropped. We will drop ‘sex_M’, ‘Medu_3’ and ‘Mjob_other’.

Keras to_categorical

Photo by Jan Antonin Kolar on Unsplash

The Keras method is a very simple method, and although it can be used for anything just like the other methods, it can only handle numeric values.

Therefore, if you have string categories you will have to convert them first, something that the other methods take care of automatically.

tf.keras.utils.to_categorical(y, num_classes=None, dtype='float32')

Documentation

Keras to_categorical() is probably most useful for one hot encoding the labels:

The above doesn’t tell us a lot so let’s pick out the transformation at index 5 so we can see what was encoded:

Reversal

There is no dedicated method for reversal, but typically argmax should allow us to reverse the encoding. Argmax will also work from the output of models where the numbers may not be whole integers.

Smaller example:

All the data:

Specify categories

One useful feature is the ability to specify how many unique categories there are. By default the amount of categories is the highest number in the array + 1. The +1 is to take account of zero.

It is worth noting that this is the minimum value you can specify. However, there may be cases where the data passed does not contain all the categories and you still wish to convert it (like a small set of test labels), in which case you should specify the number of classes.

Even though the method requires whole numbers, it can deal with the float datatype as per the below.

These are the unique classes:

This is the count of the unique classes:

There are only 18 unique classes, but we can encode as many as we want, so let’s encode 30 classes:

We can then check the shape to see that we have 30 columns / classes:

…and still reverse without issue:

Summary

As a general roundup:

Pandas — get_dummies():

creates one hot encoded columns within a dataframe without creating a new matrix. If you prefer to keep everything within a Pandas dataframe with minimal effort this may be the method for you
can only automatically recognise non-numeric columns as categorical data
has a few useful options such as sparse matrices and dropping the first column
as of Pandas 1.5.0 has a reversal method built in

Scikit-learn — OneHotEncoder():

is designed to work with Pipelines, and so is easy to integrate into your pre-processing workflow
can automatically pick out the categories for one hot encoding, including numerical columns
drop columns (not just the first, there are more extensive options available)
produce sparse matrices
various options for handling categories that appear in future datasets (handle_unknown)
you can limit the amount of categories returned from the encoding based on frequency or a maximum number of categories
has many helper methods and attributes to keep track of your encoding and parameters

Keras — to_categorical():

a very simple method that one hot encodes only numerical data
data must be converted to ordinal numerical categories first
is probably most useful for labels
has no built in reversal method

Conclusion

All in all, if I had to recommend any one method it would be the OneHotEncoder() from Scikit-Learn.

You could argue that the method is over complicated. However, I would argue that the method is very simple to use, and you ultimately gain both traceability and flexibility that is not achievable with any of the other methods.

The ability to combine this pre-processing method along with others into a processing pipeline, and features such as handle_unknown, are also a huge advantage when considering production ready code.

References

[1] Aman Chauhan, Alcohol Effects On Study (2022), Kaggle, License: Attribution 4.0 International (CC BY 4.0)

[2] Paulo Cortez, Student Performance Data Set (2014), UCI Machine Learning Repository

[3] P. Cortez and A. Silva, Using Data Mining to Predict Secondary School Student Performance (2008), In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology (FUBUTEC) Conference pp. 5–12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978–9077381–39–7

Is Julia Really Faster than Python and Numpy?

Fri, 11 Nov 2022 00:00:00 GMT

Python, along with the numpy/pandas libraries, has essentially become the language of choice for the data science profession (…I’ll add a quick nod to R here).

However, it is well known that Python, although fast and easy to implement, is a slow language. Hence the need for excellent libraries like numpy to increase efficiency…but what if there was a better alternative?

Julia claims to be at least as easy and intuitive to use as Python, whilst being significantly faster to execute. Let’s put that claim to the test…

What is Julia?

Just in case you have no idea what Julia is, here is a quick primer.

Julia is an open source language that is dynamically typed, intuitive, and easy to use like Python, but with the speed of execution of a language like C.

It has been around approximately 10 years (born in 2012), so it is a relatively new language. However, it is at a stage of maturity where you wouldn’t call it a fad.

The original creators of the language are active in a relevant field of work:

For the work we do — scientific computing, machine learning, data mining, large-scale linear algebra, distributed and parallel computing — …

julialang.org — Jeff Bezanson, Stefan Karpinski, Viral B. Shah, Alan Edelman

All in all, it is a modern language specifically designed to be used in the field of data science. The aims of the creators themselves tell you a great deal:

We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled.

(Did we mention it should be as fast as C?)

julialang.org — Jeff Bezanson, Stefan Karpinski, Viral B. Shah, Alan Edelman

Sounds quite exciting right?

The basis of the speed test

I have written an article previously looking at vectorization using the numpy library in Python:

How to Speedup Data Processing with Numpy Vectorization | Towards Data Science
Up to 8000 times faster than normal functions
Michael Clayton - Towards Data Science

The speed test that will be conducted in this article will basically be an extension / comparison to this article.

How will the test work?

A comparison will be made between the speed of execution of a simple mathematical statement:

Function 1 — Simple summation

#Python
def sum_nums(a, b):
    return a + b

#Julia
function sum_nums(x,y)
    x + y
end

and more complicated conditional statement:

Function 2 — More complex (logic and arithmetic)

#Python
def categorise(a, b):
    if a < 0:
        return a * 2 + b
    elif b < 0:
        return a + 2 * b
    else:
        return None

#Julia
function categorise(a, b)::Float32
    if a < 0
        return a * 2 + b
    elseif b < 0
        return a + 2 * b
    else
        return 0
    end
end

When run through the following methods:

Python- pandas.itertuples()
Python- list comprehension
Python- numpy.vectorize()
Python- native pandas method
Python- native numpy method
Julia- native method

The notebooks for this article

Photo by Tirachard Kumtanom

The previous article included a Jupyter notebook written in Python. I have taken this notebook (unchanged) from the previous article, and re-run it using a deepnote instance which utilises Python 3.10.

The deepnote instance for both the Python runs, and the Julia runs, has the exact same basic CPU instance (i.e. hardware). This ensures that the timed results included with this article are directly comparable.

Note: I have made sure to include the CPU information in each notebook so you can see what exact hardware was used, and that they were in fact exactly the same.

Running the Julia notebook

It is worth noting that whether you wish to use the notebooks in deepnote as I have, or in colab, you will need to setup Julia in the respective environments. This is mainly because most public online instances are currently setup for Python only (at least out of the box).

Environment Setup

Deepnote

As deepnote utilises docker instances, you can very easily setup a ‘local’ dockerfile to contain the install instructions for Julia. This means you don’t have to pollute the Jupyter notebook with install code, as you will have to do in Colab.

In the environment section select “Local ./Dockerfile”. This will open the actual Dockerfile where you should add the following:

FROM deepnote/python:3.10
RUN wget https://julialang-s3.julialang.org/bin/linux/x64/1.8/julia-1.8.2-linux-x86_64.tar.gz && \
    tar -xvzf julia-1.8.2-linux-x86_64.tar.gz && \
    mv julia-1.8.2 /usr/lib/ && \
    ln -s /usr/lib/julia-1.8.2/bin/julia /usr/bin/julia && \
    rm julia-1.8.2-linux-x86_64.tar.gz && \
    julia  -e "using Pkg;pkg\"add IJulia\""
ENV DEFAULT_KERNEL_NAME "julia-1.8"

You can update the above to the latest Julia version from this page, but at the time of writing 1.8.2 is the latest version.

Colab

For colab all the download and install code will have to be included in the notebook itself, as well as refreshing the page once the install code has run.

Fortunately, Aurélien Geron (…that name will be familiar to a few here I recon) has made available on his GitHub a starter notebook for Julia in colab, which is probably the best way to get started.

The notebooks

The raw notebooks can be found here:

notebooks/julia-python-comparison at main · thetestspecimen/notebooks
Jupyter notebooks. Contribute to thetestspecimen/notebooks development by creating an account on GitHub.
thetestspecimen - GitHub

…or get kickstarted in either deepnote or colab.

Python Notebook:

Julia Notebook:

The Results

Photo by Lukas

If you haven’t read my previous article on numpy vectorization I would encourage you to (obviously!), as it will help you get an idea of how the Python methods stack up before we jump into the Julia results.

All will be summarised and compared at the end of the article, so don’t worry too much if you don’t have the time.

The input data

Define a random number generator, and two columns of one million random numbers taken from a normal distribution, just like in the numpy vectorization article:

Function 1 — Simple summation

Using Julia’s BenchmarkTools it is possible to automatically get a fair estimate of the functions performance, as the “@benchmark” method will automatically decide how many times to evaluate the function to attain a fair estimate of runtime. It also provides a wealth of statistics as can be seen below:

To gain a fair comparison to the Python methods the mean time will be used, which in this case is 711.7 micro seconds (or 0.71 milliseconds) to sum a 1 million element array with another 1 million element array.

Function 2 — More complex (logic and arithmetic)

An example of what the method returns:

The benchmark:

So the more involved method results in a mean execution time of 7.62 milliseconds.

How does this compare to Python?

Photo by Maksim Goncharenok

Now for the actual comparison. Firstly lets see what the results look like all together:

Method	Function 1 (Simple) [ms]	Function 2 (Complex) [ms]
Python(Pandas): iter tuples	419.14	419.22
Python: list comprehension	179.64	188.33
Python(Numpy): vectorize	163.07	140.78
Python(Pandas): native	0.96	-
Python(Numpy): native	0.81	-
Julia: native	0.71	7.62

Table — All Results

Figure 1 — All Results

Then we can proceed to break it down a little.

Results: Function 1 — Simple

It is quite clear from Figure 1 for the simple summation function that there is a lot to be gained from ensuring you are using optimised libraries such as numpy where Python is concerned. The difference is so large the faster methods almost look to be zero.

I covered the reasoning for this in my previous article on numpy vectorization, so if you want more details please refer to that.

Figure 2 — Simple Function Fastest Three Results

However, even optimised Python libraries are insufficient to elevate the execution speed to the level of a language that is designed to be fast from the ground up.

As you can see in Figure 2, in this specific test Julia is 14% faster using a native inbuilt implementation, compared to an optimised library in Python (numpy) that utilises execution in C under the hood.

Results: Function 2-Complex

Although the previous result is already impressive, in the second test Julia blows the competition out of the water.

As explained in my previous article it is not possible to implement a ‘native’ version of the complex function in numpy, so we instantly lose the closest competitors from the previous round.

Even the method ‘Vectorize’ from numpy can’t hold a candle to Julia in this instance.

Figure 3— Complex Function Fastest Two Results

Julia is a full 18 times faster than numpy vectorize at completing the more complex calculation.

So what happened?

One of the ways numpy is so fast in certain circumstances, is that it is using pre-compiled and optimised C functions to execute the calculations. As you may be aware C is extremely fast if used correctly. However, the important point is that: If something is pre-compiled, then it is inherently fixed.

What this illustrates is that if your calculation is simple (like Function 1), and has a predefined optimised function within the numpy library, execution times are almost the same as Julia.

However, if the calculation you want to perform is a bit more convoluted or bespoke, and not covered by an optimised numpy function, you will be out of luck when it comes to speed. This is because you will have to rely on standard Python to fill the gap, which results in the large disparity we see in Figure 2 for the ‘complex’ function.

Conclusion

In terms of answering: Is Julia really faster than Python and Numpy?

Well, yes it is, and in some cases by a large margin.

As always, it is important to take these results for what they are, a small comparison of some specific functions. Even so, the results are real and relevant all the same. Julia is a fast language, and although we haven’t touched on it much in this article, it is actually very intuative to use as well.

If you want a more general guide as to how fast Julia is compared to a wider array of languages then you can take a look at the general benchmarks on Julia’s own website:

Julia Micro-Benchmarks
The official website for the Julia Language. Julia is a language that is fast, dynamic, easy to use, and open source. Click here to learn more.
Jeff Bezanson, Stefan Karpinski, Viral Shah, Alan Edelman, et al. - julialang.org

Same result, it is in a league of it’s own, especially in the data science world. Unless of course you write all your code in C.

A final question…

Image by Arek Socha from Pixabay

If Julia is so good, why doesn’t it have the same traction and recognition as Python/Pandas/NumPy/R?

I think the answer to that question is mainly time. It just hasn’t been around long enough, but the reality is that Julia is on the up, and at some point it is likely (at least in my opinion) that it will take over in the field of data science. It is already being used by the likes of Microsoft, Google, Amazon, Intel, IBM and Nasa, for example.

Python and R are industry standards at this stage, and that will take a lot of momentum to change, regardless of how good the new upstart is.

A additional factor is the availability of learning resources. Again, just due to time, and the sheer volume of people using Python and R for data science, resources to learn from are plentiful. Whereas for Julia although there is plenty of documentation, it can’t really compete for overall learning resources (yet!).

…but if you are feeling adventurous I encourage you to give Julia a go and see what you think.

Julia’s Flux vs Python’s TensorFlow - How Do They Compare?

Fri, 02 Dec 2022 00:00:00 GMT

In my previous article I looked at what sort of advantage Julia has over Python/Numpy in terms of speed.

Although this is useful to know, it isn’t the whole story. It is also important to understand how they compare in terms of syntax, library availability / integration, flexibility, documentation, community support etc.

This article runs though an image classification deep learning problem from start to finish in both TensorFlow and Flux (Julia’s native TensorFlow equivalent). This should give a good overview of how the two languages compare in general usage, and hopefully help you get an insight into whether Julia is a potential option (or advantage) for you in this context.

I will also endeavour to highlight the advantages, and more importantly the gaps, or failings, that currently exist in the Julia ecosystem when compared to the tried and tested pairing of Python and TensorFlow.

Introduction

I have deliberately chosen an image classification problem for this particular exploration, as it throws up some nice challenges in both data preparation, and for the deep learning frameworks themselves:

the images need to be loaded from disk (a ready prepared dataset such as MNIST will not be used), so loading and pre-processing methods and conventions will be explored
images are typically represented as 3D-matrices (height, width, colour channels), so careful attention to dimensional ordering will be required
to avoid over-fitting, image augmentation is typically required, allowing for an exploration into library availability and ease of use
images are inherently a ‘large’ data type in terms of space requirements, which forces an investigation into batching, RAM allocation and GPU usage

Note: Although code snippets will be made available throughout the article, Jupyter notebooks are available containing a full working end-to-end (image download through to model training) implementation of both the Julia and Python versions of the code. See the next section for links to the notebooks.

Incidentally, if this is the first time you have heard of Julia I recommend reading the “What is Julia?” section of my previous article to get a quick primer:

Is Julia Really Faster than Python and Numpy? | Towards Data Science
The speed of C with the simplicity of Python
Michael Clayton - Towards Data Science

Reference Notebooks

This section gives details on the location of the notebooks, and also the requirements for the environment setup for online environments such as Colab and Deepnote.

The notebooks

The raw notebooks can be found here for your local environment:

notebooks/julia-python-image-classification at main · thetestspecimen/notebooks
Jupyter notebooks. Contribute to thetestspecimen/notebooks development by creating an account on GitHub.
thetestspecimen - GitHub

…or get kickstarted in either deepnote or colab.

Python Notebook:

Julia Notebook:

Environment Setup for Julia

Deepnote

In the environment section select “Local ./Dockerfile”. This will open the actual Dockerfile where you should add the following:

FROM deepnote/python:3.10

RUN wget https://julialang-s3.julialang.org/bin/linux/x64/1.8/julia-1.8.3-linux-x86_64.tar.gz && \
    tar -xvzf julia-1.8.3-linux-x86_64.tar.gz && \
    mv julia-1.8.3 /usr/lib/ && \
    ln -s /usr/lib/julia-1.8.3/bin/julia /usr/bin/julia && \
    rm julia-1.8.3-linux-x86_64.tar.gz && \
    julia  -e "using Pkg;pkg\"add IJulia\""

ENV DEFAULT_KERNEL_NAME "julia-1.8"

You can update the above to the latest Julia version from this page, but at the time of writing 1.8.3 is the latest version.

Colab

For colab all the download and install code will have to be included in the notebook itself, as well as refreshing the page once the install code has run.

Fortunately, Aurélien Geron has made available on his GitHub a starter notebook for Julia in colab, which is probably the best way to get started.

Note: if you use the “Open in Colab” button above (or the Julia notebook ending in “colab” from the repository I linked) I have already included this starter code in the Julia notebook.

The Data

The data^[1] utilised in this article is a set of images which depict the three possible combinations of hand position used in the game rock-paper-scissors.

Four examples from the three different categories of the dataset. Composite image by Author.

Each image is of type PNG, and of dimensions 300(W) pixels x 200(H) pixels, in full colour.

The original dataset contains 2188 images in total, but for this article a smaller selection has been used, which comprises of precisely 200 images for each of the three categories (600 images in total). This is mainly to ensure that the notebooks can be run with relative ease, and that the dataset is balanced.

The smaller dataset that was used in this article is available here:

notebooks/datasets/rock_paper_scissors at main · thetestspecimen/notebooks
Jupyter notebooks. Contribute to thetestspecimen/notebooks development by creating an account on GitHub.
thetestspecimen - GitHub

The Plan

There are two separate notebooks. One written in Python using the TensorFlow deep learning framework, and a second that is written in Julia and utilises the Flux deep learning framework.

Both notebooks use exactly the same raw data, and will go through the same steps to finally end up with a trained model at the end.

Although it is not possible to match the methodology exactly between the two notebooks (as you might expect), I have tried to keep them as close as possible.

A general outline

Each notebook covers the following steps:

Download the image data from a remote location and extract into local storage
Load the images from a local folder structure ready for processing
Review image data, and view sample images
Split the data into train / validation sets
Augment the training images to avoid over-fitting
Prepare the images for the model (scaling etc.)
Batch data
Create model and associated parameters
Train model (should be able to use the CPU or GPU)

Comparison — Packages

Photo by Leone Venter on Unsplash

The comparison sections that follow will explore some of the differences (good or bad) that Julia has with the Python implementation. It will generally be broken down as per the bullet points in the previous section.

Package installation

To start with, a quick note on package installation and usage.

The two languages follow a similar pattern:

make sure the package is installed in your environment
‘import’ the package into your code to use it

The only real difference is that Julia can either install packages in the ‘environment’ before running code, or the packages can be installed from within the code (as is done in the Julia notebook for this article):

“Pkg” in Julia is the equivalent of “pip” in Python, and can also be accessed from Julia’s command line interface.

An example of adding a package using the command line. Screenshot by author

Package usage

In terms of being able to access the installed packages from within the code, you would typically use the keyword “using” rather than “import”, as used in Python:

Julia does also have an “import” keyword too. For more details on the difference please take a look at the documentation. In most general use cases “using” is more appropriate.

Note: in this article I have deliberately used the full path, including the module names, when referencing a method from a module. This is not necessary, it just makes it clearer which package the methods are being referenced from. For example these two are equivalent and valid:

using Random

Random.shuffle(my_array) # Full path

shuffle(my_array) # Without package name

Comparison — Download and Extraction

Photo by Miguel Á. Padriñán on Pexels

The image data is provided remotely as a zip file. The zip file contains a folder for each of the three categories, and each folder has 200 images contained within.

First step, download and extract the image files. This is relatively easy in both languages, and with the available libraries, probably more intuitive in Julia. The Python implementation:

What the Julia implementation could look like:

using InfoZIP

download("https://github.com/thetestspecimen/notebooks/raw/main/datasets/rock_paper_scissors/rock_paper_scissors.zip","./rock_paper_scissors.zip")

root_folder = "rps"
isdir("./$root_folder") || mkdir("./$root_folder")

InfoZIP.unzip("./rock_paper_scissors.zip", "./$root_folder")

You may notice that I said what it it could look like. If you look at the notebooks you will see that in Julia I have actually used a custom function to do the unzipping, rather than utilising the InfoZIP package as detailed above.

The reason for this is that I couldn’t get the InfoZIP package to install in all of the environments I used, so I thought it would be unfair to include it.

Is this a fault of the package? I suspect not. I think this is likely due to the fact that the online environments (colab and deepnote) are not primarily geared towards Julia, and sometimes that can cause problems. InfoZIP installs and works fine locally.

Is this a fault of the package? I suspect not.

I should also note here that to use the “Plots” library (which is the equivalent of something like matplotlib) in colab when utilising a GPU instance will result in failure to install!

This may seem relatively trivial, but it is a direct illustration of the potential problems you can run into when dealing with a new-ish language. Furthermore, you are less likely to find a solution to the problem online as the community is smaller.

It is still worth pointing out that when working locally on my computer I had no such issues with either package, and I would hope that with time online environments such as colab and deepnote may be a bit more Julia friendly out of the box.

Comparison — Handling images

Photo by Ivan Shimko on Unsplash

Now the images are in the local environment, it is possible to take a look at how we can load and interact with them.

Python:

Julia:

In the previous section things were relatively similar. Now things start to diverge…

This simple example of loading an image highlights some stark differences between Python and Julia, although that might not be immediately obvious from two similar looking code blocks.

Zero indexing vs one indexing

Probably one of the major differences between the languages in general, is the fact that Julia uses indexing for arrays and matrices starting from 1 rather than 0.

Python — the first element of “random_image”:

img = mpimg.imread(target_folder + "/" + random_image[0])

Julia — the first element of the shape of “image_channels”:

println("Colour channels: ", size(img_channels)[1])

I can see how this could be a contentious difference. It all comes down to preference in reality, but putting any personal preferences aside, 1-indexing makes a lot more sense mathematically.

It is also worth remembering the field of work this language is aimed at (i.e. more mathematical / statistics based professionals who are programming, rather than pure programmers / software engineers).

Regardless of your stance, something to be very aware of, especially if you are thinking of porting a project over to Julia from Python.

How images are loaded and represented numerically

When images are loaded using python and “imread” they are loaded into a numpy array with the shape (height, width, RGB colour channels) as float32 numbers. Pretty simple.

In Julia, images are loaded as:

Type: Matrix{RGB{N0f8}}

Not so clear…so let’s explore this a little.

In JuliaImages, by default all images are displayed assuming that 0 means “black” and 1 means “white” or “saturated” (the latter applying to channels of an RGB image).

Perhaps surprisingly, this 0-to-1 convention applies even when the intensities are encoded using only 8-bits per color channel. JuliaImages uses a special type, N0f8, that interprets an 8-bit "integer" as if it had been scaled by 1/255, thus encoding values from 0 to 1 in 256 steps.

-Juliaimages.org

Turns out this strange convention used by Julia is actually really helpful for machine / deep learning.

One of the things you regularly have to do when dealing with images in Python is to scale the values by 1/255, so that all the values fall between 0 and 1. This is not necessary with Julia, as the scaling is automatically done by the “N0f8” type used for images natively!

Unfortunately, you will not see the comparison in this article as the images are type PNG, and imread in Python returns the array as float values between 0 and 1 anyway (incidentally the only format it does that for).

Photo by Michael Maasen on Unsplash

However, if you were to load in a JPG file in Python, you would receive integer values between 0 and 255 as the output from imread, and have to scale them later in your pre-processing pipeline.

Apart from the auto scaling, it is also worth noting how the image is actually stored. Julia uses a concept of representing each pixel as a type of object, so if we look at the output type and shape:

Type: Matrix{RGB{N0f8}}
Shape: (200, 300)

It is literally stated as a 200x300 matrix. What about the three colour channels? We would expect 200x300x3, right?

Well, Julia views each of the items in the 200x300 matrix as a ‘pixel’, which in this case has three values representing Red, Green and Blue (RGB), as indicated in the type ‘RGB{N0f8}’. I suppose it would be like a matrix of objects, the object being defined as having three variables.

However, there is reason behind the madness:

This design choice facilitates generic code that can handle both grayscale and color images without needing to introduce extra loops or checks for a color dimension. It also provides more rational support for 3d grayscale images–which might happen to have size 3 along the third dimension–and consequently helps unify the “computer vision” and “biomedical image processing” communities.

-Juliaimages.org

Some real thought went into these decisions it would seem.

However, you cannot feed an image in this format into a neural network, not even Flux, so it will need to be split out into a ‘proper’ 3D matrix at a later stage. Which as it turns out is very easy indeed, as you will see.

Comparison — Data preparation pipeline

Photo by JJ Ying on Unsplash

This is definitely one of the areas where, in terms of pure ease of use Python and TensorFlow blow Julia out of the water.

The Python implementation

I can more or less load all my images into a batched optimised dataset ready to throw into a deep learning model in essentially four lines of code:

Training image augmentation is taken care of just as easily with a model layer:

The Julia implementation

To achieve the same thing in Julia requires quite a bit more code. Let’s load the images and split them into train and validation sets:

I should note that in reality, you could lose the “load and scale” and “shuffle” sections of this method and deal with it in a more terse form later, so it isn’t as bad as it looks. I mainly left those sections in as a reference.

One useful difference between Python and Julia is that you can define types if you want, but it is not absolutely necessary. A good example is the type of “Tuple{Int,Int}” specified for the “image_size” in the function above. This ensures whole numbers are always passed, without having to do any specific checking within the function itself.

Augmentation pipeline

Image augmentation is very simple just like TensorFlow thanks to the Augmentor package, you can also add the image resizing here using the “Resize” layer (a more terse form as alluded to earlier):

You should also note that Augmentor (using a wrapper of Julia Images) has the ability to change the ‘RGB{N0f8}’ type to a 3D matrix of type float32 ready for passing into the deep learning model:

I want to break down the three steps above as I think it is important to understand what exactly they do, as I can see it is potentially quite confusing:

SplitChannels — takes an input of Matrix{RGB{N0f8}} with shape 160 (height) x 160 (width) and converts it to 3 (colour channels) × 160 (height) × 160 (width) with type Array{N0f8}. It is worth noting here that the colour channels become the first dimension, not the last like in python/numpy.
PermuteDims — just rearranges the shape of the array. In our case we change the dimensions of the output to 160 (width) x 160 (height) x 3 (colour channels). Note: the order of height and width have also been switched.
ConvertEltype — changes N0f8 into float32.

You may be wondering why the dimensions need to be switched about so much. The reason for this is due to the requirement of the input shape into a Conv layer in Flux at a later stage. I will go into more detail once we have completed batching, as it one of my main gripes with the whole process…

Applying the augmentation

Now we have an interesting situation. I need to apply the augmentation pipelines to the images. No problem! Due to the excellent augmentbatch!() function provided by the Augmentor package.

Except no matter how hard I tried I couldn’t get it to work with the data. Constant errors (forgot to note down exactly what while I was frantically trying to sort out a solution, but there were similar problems in various forums).

There is always the possibility that this is my fault, I kind of hope it is!

As a workaround, I used a loop using the ‘non-batched’ method augment. I also one-hot encoded the labels at the same time using the OneHotArrays package:

I think this goes a long way to illustrate that there are some areas of the Julia ecosystem that may give you a few implementation headaches. You may not always find a solution online either, as the community is smaller.

However, one of the major advantages of Julia is that if you have to resort to things like a for loop to navigate a problem, or just to implement a bit of a bespoke requirement, you can be fairly sure that the code you write will be optimised and quick. Not something you can rely on every time with Python.

Some example augmented images from the train pipeline:

Batching the data

Batching in Julia is TensorFlow level easy. There are multiple ways of going about this in Julia. In this case Flux’s built in DataLoader will be used:

Nothing much to note about the method. Does exactly what it says on the tin. (This is the alternative place you can shuffle the data as I alluded to earlier.)

We are ready to pass the data to the model, but first a slight detour…

Confusing input shapes

Photo by Torsten Dederichs on Unsplash

I now want to come back to the input shape for the model. You can see in the last code block of the previous section that the dataset has shape:

Data: 160(width) x 160(height) x 3(colour channels) x 32(batchsize)
Labels: 3(labels) x 32(batchsize)

In TensorFlow the input shape would be:

Data: 32(batchsize) x 160(height) x 160(width) x 3(colour channels)
Labels: 32(batchsize) x 3(labels)

From the Julia documentation:

Image data should be stored in WHCN order (width, height, channels, batch). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

-fluxml.ai

I have no idea why this convention has been chosen, especially the switching of height and width.

Frankly, there is nothing to complain about, it is just a convention. In fact, if previous experience is anything to go by I would suspect some very well thought out optimisation is the cause.

I have come across some slightly odd changes in definition compared to other languages, only to find out there is a very real reason for it (as you would hope).

As a concrete (and relevant) example, take Julia’s Images package.

The reason we use CHW (i.e., channel-height-width) order instead of HWC is that this provides a memory friendly indexing mechanism for Array. By default, in Julia the first index is also the fastest (i.e., has adjacent storage in memory). For more details, please refer to the performance tip: Access arrays in memory order, along columns

-juliaimages.org

Extra confusion

This brings up one of the main gripes I have.

I can accept that it is a different language, so there is potentially a good reason to have a new convention. However, during the course of loading images and getting them ready for plugging into a deep learning model in Flux, I have had to morph the input shape all over the place:

Images are loaded in: Matrix{RGB{N0f8}} (height x width)
Split channels: {N0f8} (channels x height x width)
Move channels AND switch height and width: {N0f8} (width x height x channels)
Convert to float32
Batchsize (as last element): {float32} (width x height x channels x batchsize)

As apparently (channels x height x width) is optimal for images, and that is how native Julia loads them in. Could it not be:

(channels x height x width x batchsize)?

Might save a lot of potential confusion.

I genuinely hope that someone can point out that I have stupidly overlooked something (really I do). Mainly because I have been very impressed with the attention to detail and thought that has gone into designing this language.

OK. Enough moaning. Back to the project.

Comparison — The model

Photo by DS stories on Pexels

Model definition in Julia is very similar to the Sequential method of TensorFlow. It is just called Chain instead.

Note: I will mostly cease to include the TensorFlow code in the article at this point just to keep it readable, but both notebooks are complete and easy to reference if you need to.

Main differences:

You must explicitly load both the model (and data) onto the GPU if you want to use it
You must specify the input and output channels explicitly (input=>output) — I believe there are shape inference macros that can help with this, but we won’t get into that here

All in all very intuitive to use. It also forces you to have a proper understanding of how the data moves and reshapes through the model. A good practise if you ask me.

There is nothing more dangerous that a black box system, and no thinking. We all do it of course as sometimes we just want the answer quickly, but it can lead to some hard to trace and confusing outcomes.

There is nothing more dangerous than a black box system, and no thinking. We all do it…

Calculation device specification

In the code for the notebook you can see where I have explicitly defined which device (CPU or GPU) should be used for calculation by using the variable “calc_device”.

Changing the “calc_device” variable to gpu will use the gpu. Change it to cpu to use only the cpu. You can of course replace all the “calc_device” variables with gpu or cpu directly, and it will work in exactly the same way.

Comparison — Loss, Accuracy and Optimiser

Image by 3D Animation Production Company from Pixabay

Again, very similar to TensorFlow:

A couple of things of note:

logitcrossentropy

You may have noted that the model has no softmax layer (if not take a quick look).

This is mathematically equivalent to crossentropy(softmax(ŷ), y), but is more numerically stable than using functions crossentropy and softmax separately.

-fluxml.ai

onecold

The opposite of onehot. Excellent name, not sure why someone hasn’t thought of that before.

One line function definition

If you are new to Julia it is also worth pointing out that the loss and accuracy functions are actually proper function definitions in one line. i.e. the same as this:

function loss(X, y)
  return Flux.Losses.logitcrossentropy(model(X), y)
end

One of many great features of the Julia language.

(Note: you could actually omit the return keyword in the above. Another way of shortening a function.)

Comparison — Training

Photo by Victor Freitas on Unsplash

Training can be as complicated or simple as you like in Julia. There is genuinely a lot of flexibility available, I haven’t even touched the surface in what I’m about to show you.

If you want to keep it really simple there is what I would call the equivalent to “model.fit” in TensorFlow:

for epoch in 1:10
    Flux.train!(loss, Flux.params(model), train_batches, opt)
end

That’s right, basically a for loop. You can also add a callback parameter to do things like print loss or accuracy (which is not done automatically like TensorFlow), or early stopping etc.

However, using the above method can cause problems when dealing with large amounts of data (like images), as it requires loading all of the train data into memory (either locally or on the GPU).

The following function therefore allows the batches to be loaded onto the GPU (or memory for a CPU run) batch by batch. It also prints the training loss and accuracy (an average over all batches), and the validation loss on the whole validation dataset.

Note: If the validation set is quite large you could also calculate the validation loss/accuracy on a batch by batch basis to save memory.

In reality, it is two for loops: one for the epochs and an internal loop for the batches.

The lines of note are:

x, y = device(batch_data), device(batch_labels)
gradients = Flux.gradient(() -> loss(x, y), Flux.params(model))
Flux.Optimise.update!(optimiser, Flux.params(model), gradients)

This is the loading of a batch of data onto the device (cpu or gpu), and running the data through the model.

All the rest of the code is statistics collection and printing.

More involved than TensorFlow, but nothing extreme.

…and we are done. An interesting journey.

Summary (and TL;DR)

Photo by Ann H on Pexels

At the beginning of the article I stated that apart from speed there are other important metrics when it comes to deciding if a language is worth the investment compared to what you already use. I specifically named:

syntax
flexibility
library availability / integration
documentation
community support

Having been over a whole project I thought it might be an idea to summarise some of the findings. Just bear in mind that this is based on my impressions while generating the code for this article, and is only my opinion.

Syntax

Coming from Python I think that the syntax is similar enough that it is relatively easy to pick up, and I would suggest that in quite a few circumstances it is even more ‘high level’ and easy to use than Python.

Let’s get the contentious one out of the way first. Yes, Julia uses 1-indexed arrays rather than 0-indexed arrays. I personally prefer this, but I suspect there will be plenty who won’t. There are also more subtle differences such as array slicing being inclusive of the last element, unlike Python. Just be a little careful!

But there is plenty of good stuff…

For example when using functions you don’t need a colon or return keyword. You can even make a succinct one liner without losing the codes meaning:

# this returns the value of calculation, no return keyword needed

function my_func(x , y)
      x * y + 2
end

# you can shorten this even further

my_func(x, y) = x * y + 2

Notice the use of “end” in the standard function expression. This is used as indentation doesn’t matter in Julia, which in my opinion is a vast improvement. The spaces vs tabs saga will not apply to Julia.

The common if-else type statements can also be utilised in very terse and clear one line statements:

# ternary - if a is less than b print(a) otherwise print(b)

(a < b) ? print(a) : print(b)

# only need to do something if the condition is met (or not met)? 
# use short circuit evaluation.

# if a is less than b print(a+b), otherwise do nothing

(a < b) && print(a+b)

# if a is less than b do nothing, otherwise print(a+b)

(a < b) || print(a+b)

I’m likely just scratching the surface here, but already I am impressed.

Flexibility

I think flexibility is something that Julia really excels at.

As I have already mentioned, you can write code that is terse and to the point just like Python, but there are also additional features should you need, or want, to utilise them.

The first and foremost is probably the option to use types, something not possible in Python. Although inferred types sound like a great idea, they do have various downsides, such as making code harder to read and follow, and introducing hard to trace bugs.

flexibility to specify types when it makes the most sense is an excellent ability

Having the flexibility to specify types when it makes the most sense is an excellent ability that Julia has. I’m glad it isn’t forced across the board though.

Julia is also aimed at scientific and mathematical communities. Utilising unicode characters in your code, is therefore quite a useful feature. Not something I will likely use, but as I come from a mathematical / engineering background I can appreciate the inclusion.

Library availability / consistency

This is a bit of a mixed bag.

Photo by Iñaki del Olmo on Unsplash

This article has utilised quite a few packages. From some larger packages such as Flux and Images, right down to more bespoke packages such as OneHotArrays and Augmentor.

On the whole I would say they don’t, on average, approach the level of sophistication, integration, and ease of use that you can find in Python / TensorFlow. It takes a little bit more effort do the same thing, and you are likely to find more problems, and hit more inconsistencies. I am not surprised by this, at the end of the day it is a less mature ecosystem.

For example the ability to batch and optimise your data with a simple one line interface is a really nice feature of TensorFlow. The fact you don’t have to write extra code to print training and validation loss / accuracy is also very useful.

However, I think Julia’s library ecosystem has enough variation and sophistication to genuinely do more than enough. The packages on the whole play nicely together too. I don’t think it is even close to a deal breaker.

To summarise my main issues I encountered with the packages in this article:

I couldn’t get the package InfoZIP to install consistently across all environments
I couldn’t get the !augmentbatch() function to work for the data in this article at all, which would have been useful
For some reason there is a slightly muddled approach to how the shape of an image is defined between JuliaImages and Flux, which leads to quite a lot of messing about with re-shaping matrices. It isn’t hard, it just seems unnecessary.

Documentation

Documentation for the core language is a pretty complete and reliable source. The only thing I would suggest is that the examples given for some methods could be a bit more detailed and varied. A minor quibble, otherwise excellent stuff.

Moving beyond the core language, and the detail and availability of the documentation can vary.

I’m impressed with the larger packages, which I suppose would be almost core packages anyway. In terms of this article, that would be JuliaImages and Flux. I would say they are pretty comprehensive, and I particularly like the effort that goes into emphasising why things are done a certain way:

The reason we use CHW (i.e., channel-height-width) order instead of HWC is that this provides a memory friendly indexing mechanisim for Array. By default, in Julia the first index is also the fastest (i.e., has adjacent storage in memory).

-juliaimages.org

As the packages get smaller, the documenation is generally there, but a bit terse. ZipFile is a good example of this.

Although, typically packages are open source and hosted on Github, and contributions are usually always welcome. As stated by JuliaImages:

Please help improve this documentation–if something confuses you, chances are you’re not alone. It’s easy to do as you read along: just click on the “Edit on GitHub” link above, and then edit the files directly in your browser. Your changes will be vetted by developers before becoming permanent, so don’t worry about whether you might say something wrong.

-juliaimages.org

Community

The community is exactly what I expected it to be, lively and enthusiastic, but significantly smaller than Python’s / TensorFlow’s. If you need answers to queries, especially more bespoke queries, you may need to dig a little deeper than you usually would into the likes of Google and StackOverflow.

This will obviously change with adoption, but fortunately the documentation is pretty good.

Conclusion

All things considered I think Julia is a great language to actually use.

The creators of the language were trying to take the best parts of the languages they loved to use, and combine them into a sort of super language, which they called Julia.

In my opinion they have done an extremely good job. The syntax is genuinely easy to use and understand, but can also incorporate advanced and slightly more obscure elements — it is a very flexible language.

It is also genuinely fast.

Image by Wokandapix from Pixabay

Yes, there may be a slight learning curve to switch from the language you are using at the moment, but again, I don’t think it will be as severe as you might think. To help you out, Julia’s documentation includes a nice point by point comparison for major languages:

Noteworthy Differences from other Languages · The Julia Language
Documentation for The Julia Language.
null - julialang.org

The only downsides I see are due to the fact that it is, even after 10 year of existence, relatively new compared to it’s competitors. This has a direct impact on the quality and quantity of documentation and learning resources. Which appears to have a bigger impact on adoption than most people would like to admit. Money also helps, as always…but I don’t have the data to comment on that situation.

Being the best product or solution does not guarantee success and broad acceptance. That is just not how the real world works.

…but after getting to know how Julia works (even on a basic level) I do hope more people see the potential and jump onboard.

References

[1] Julien de la Bruère-Terreault, Rock-Paper-Scissors Images (2018), Kaggle, License: CC BY-SA 4.0

A Simple Way to Speed Up Your Python Code — Stay Up to Date

Tue, 10 Jan 2023 00:00:00 GMT

A lot of time and money is spent trying to optimise code so that it can be as fast and efficient as possible, and within the field of data science this is becoming even more important due to the vast datasets that are now required to be processed.

A simple, and often overlooked, way of achieving code optimisation is just to make sure that your language of choice, and associated libraries, are as up to date as they can reasonably be.

You will probably be surprised how relatively little time and effort, can result in some significant benefits.

Introduction

It is a fairly obvious statement to say that staying up to date could help “optimise” your code, but blindly updating your software or libraries without understanding what is changing is potentially a recipe for disaster.

By the end of this article you should have a good idea of why you should stay up to date. You will have a solid plan of action to ensure you can hit a balance between ensuring you are optimised, and not wasting valuable time. In addition, you will also be aware of the potential pitfalls that can arise, and how to avoid them.

To round things off, a concrete example using the latest release of NumPy (1.24.0) to illustrate the real world benefits of keeping your software and libraries bang up to date.

Why should I stay up to date?

Photo by Ann H from Pexels

The simple answer is that you can benefit from items such as:

brand new features
optimisations
bug fixes

All implemented for you by people who know software well, because remember…

You are not a software development expert

If you are a data science professional (or enthusiast), which I’m going to assume you are, then your main concern is processing, manipulating and analysing data to gain insights and predictions.

Although you may have a certain level of competence with regard to software development, and general coding, it would be fair to say that it is not your expertise.

As such, it is perfectly reasonable that you rely on a high level intuitive language (Python, R, Matlab etc.), and a mountain of libraries that provide a whole host of functionality and optimisation relevant to your field of work. Allowing you to concentrate on your profession.

Rely on the experts, they know better

If you can build optimisation into your code, great! But it should not be the thing that is consuming your time.

As it turns out, there is an army of other professionals, who know software development extremely well. They are working hard to ensure the language and libraries you use are optimised, and constantly improved. Providing you with the exact tools you need to apply to your work.

However, to take advantage of these optimisations you need to pay attention, or you may be missing out.

Read the release notes, it’s important

Photo by Pixabay

New versions of both the language, and associated libraries, are released with surprising regularity. However, if you don’t pay attention to what is actually changing, then you may miss out on the potential benefits, or introduce bugs / nonsensical parameters.

A good example of both of these situations is an algorithm update applied to the NumPy np.in1d function in NumPy 1.24.0, which is also utilised in the widely used np.isin function.

np.in1d (used by np.isin) can now switch to a faster algorithm (up to >10x faster) when it is passed two integer arrays. This is often automatically used, but you can use kind="sort" or kind="table" to force the old or new method, respectively.

-numpy.org

Missing out on gains

It is important to note in the quote above that kind="table" is “often automatically used”, which implies not always.

You could be missing out on a 10x speed gain

You could be missing out on a 10x speed gain just because you didn’t meet the requirements set by the developers to automatically use the new method:

If None, will automatically choose ‘table’ if the required memory allocation is less than or equal to 6 times the sum of the sizes of ar1 and ar2, otherwise will use ‘sort’. This is done to not use a large amount of memory by default, even though ‘table’ may be faster in most cases.

-numpy.org

If you read the release notes, then simply adding kind=”table” in the method parameters would be the simple change you needed to ensure a significant benefit in speed of execution, regardless of memory allocation.

Note: this automatic selection is likely introduced to ensure the new method doesn’t cause bugs in your code. There are cases where it could use more memory than the previous method, which may be a problem in memory restricted environments. Very sensible edge case coverage by the developers.

Introducing bugs or confusing code

If you don’t set the new parameter (because you didn’t read the release notes), then very sensibly the developers set a default.

If None, will automatically choose ‘table’ if the required memory allocation is less than or equal to 6 times the sum of the sizes of ar1 and ar2.

-numpy.org

Great! So you may automatically get a speed up of 10x. However:

If ‘table’ is chosen, assume_unique will have no effect.

-numpy.org

In this case “assume_unique” having no effect is a minor problem, as the developers look like they have covered edge cases to ensure that if you previously assigned “assume_unique” it will just be ignored.

However, it is messy, as your code specifies a parameter that is irrelevant, which could cause confusion for other people (or even you) in the future.

It is also worth noting that there may be cases where the developers weren’t so thorough, or breaking changes couldn’t be avoided. If that happens, then you will suddenly have code that doesn’t run.

New methods

Apart from potentially missing out on the improvement for the methods you already use in your code, you may miss out on completely new methods that would be of potential use to your project or workflow.

In a previous article about one hot encoding data I touched on a new method implemented in the recently released Pandas 1.5.0 called from_dummies(), which is a reversal for get_dummies(), a commonly used one hot encoding method from Pandas:

Page not found | Towards Data Science
Publish AI, ML & data-science insights to a global community of data professionals.
Partha Sarkar - Towards Data Science

Prior to the release of this method, reversal of the one hot encoding would have been a manual procedure.

There are a whole host of new methods created across the libraries that you use, but the only way to be aware of when they appear is to review the release notes.

whatever method the developers have implemented will be optimised and less prone to bugs

…and remember, it is likely that whatever method the developers have implemented will be optimised and less prone to bugs than something you have coded yourself. They are professional software developers after all, and producing bug free well tested code is no easy feat.

A strategy to keep up to date

Photo by Patrick Perkins on Unsplash

Time is money, as the famous aphorism goes.

The reality is that a typical project may include a mountain of libraries, so it may not be practical to read the release notes for every library, for every tiny update, so it may be wise to prioritise.

Update on specific release points

It is quite common to see numbering conventions that include three numbers separated by dots. For example the current release of NumPy is 1.24.0.

Being aware of what these numbers (generally) signify can help plan when to pay attention:

Major version number (1.24.0): large and significant changes to the software or library. Can include changes that are not backward compatible. This requires careful review before upgrading.
Minor version number (1.24.0): typically a minor feature change / changes, or a larger set of bug fixes.
Patch version number (1.24.0): typically a smaller set of bug fixes.

If the major version number changes it is absolutely essential that you review the release notes, as there could be breaking changes (i.e. your code could stop working completely in some cases).

The minor version number change is something you should be paying attention to, as it is optimal in terms potential gains, and new or improved methods.

The patch version number can generally be passed over if you don’t have time, unless you are waiting for a known bug to be fixed.

Pick the most important libraries

If time is of the essence, you should pick the libraries that are most utilised, or most relevant to the project (i.e. those with the biggest impact), and keep up to speed with any relevant changes as updates arrive. As detailed in the previous section this should involve mainly paying attention to major and minor version number changes.

It is not essential that you update the library for every patch release, but keeping track of the changes that are published will give you the opportunity to update when it is beneficial to the project.

In reality, the smaller the update you apply, the easier it is to pin down bugs when they do occur, reducing time taken for mitigation, and lessening the risk involved.

Update strategy - Summary

Pay attention to major version number changes for ALL libraries.
Pay attention to minor version number changes for libraries that are most utilised, or most relevant to the project
Patch version number changes can mostly be passed over without review, unless you are actively waiting for a bug to be fixed

The potential dangers of upgrading

Photo by Raúl Nájera on Unsplash

As already discussed, there are some real gains to be had from keeping up to date, but it can also cause unwanted problems, and lost time, due to debugging new errors.

Again, this can mostly be avoided by paying attention to the release notes.

I will refer you to the release notes of NumPy for version 1.24.0 as an example of what to expect. It is well laid out with lots of information (not always the case with all libraries/software):

The main problems that you will face are generally due to one of the following.

Deprecations / Expired Deprecations

This is probably the main item to pay attention to. When a method becomes deprecated it will eventually be removed (timescales vary). This is excellent as, assuming you are aware of the deprecation being applied, it gives you time to implement a work around, or switch to an updated method from the library.

However, if you are looking at the release notes and notice one of your methods is listed in expired deprecations, then an immediate fix is required before upgrade, as your code will literally stop working.

Compatibility notes / changes

This is less severe than a deprecation, but can be tricky as they can change behaviour and/or outputs of functions. Something you may not immediately notice if you weren’t aware of it.

array.fill(scalar) may behave slightly different

numpy.ndarray.fill may in some cases behave slightly different now due to the fact that the logic is aligned with item assignment.

Previously casting may have produced slightly different answers when using values that could not be represented in the target dtype or when the target had object dtype.

-numpy.org

“May behave slightly differently” is about as vague as it gets! A perfect example of something that could cause ‘interesting’ bugs if it is relevant to your code. The above would be worth looking into should you have a very sensitive implementation that uses this function.

What if I can’t upgrade?

Photo by Towfiqu barbhuiya from Pexels

In industry the reality is that you may have limitations on the upgrade process that are outside of your control. This could be caused by a choice of base operating system / container, or due to limitations in other areas of the project you are working on that require very specific versioning.

Remember, without solid information, no decisions can be made.

However, it is still worth keeping up to date with the developments of key packages and libraries. If nothing else it is ammunition to push for what might be quite a complicated, and/or costly, upgrade process to be put into action, as the benefits may outweigh the upgrade cost. Remember, without solid information, no real decisions can be made.

A concrete example of the benefits of staying up to date

Photo by Rodolfo Quirós from Pexels

Fairly recently, NumPy had a minor version number change from 1.23.5 to 1.24.0. As explained in an earlier section of this article, a minor version number change, can result in “minor feature changes”, rather than just the simple bug fixes that a patch update would bring.

A couple of these “minor feature changes” claim to result in significantly sped up versions of their original functions:

np.in1d

np.in1d (used by np.isin) can now switch to a faster algorithm (up to >10x faster)

-numpy.org

NumPy comparison functions

The comparison functions (numpy.equal, numpy.not_equal, numpy.less, numpy.less_equal, numpy.greater and numpy.greater_equal) are now much faster as they are now vectorized with universal intrinsics. For a CPU with SIMD extension AVX512BW, the performance gain is up to 2.57x, 1.65x and 19.15x for integer, float and boolean data types, respectively (with N=50000).

-numpy.org

Testing the claims

Although I have no reason to doubt the numbers, I thought it might be interesting to test it out in practise, so the following sections will run benchmarks between the old an new versions to see what real world speed gains can be realised.

Notebooks

All the code that follows is available in jupyter notebooks.

This section gives details on the location of the notebooks, and also the requirements for the environment setup for online environments such as Colab and Deepnote.

The raw notebooks can be found here for your local environment:

notebooks/python-libraries-update at main · thetestspecimen/notebooks
Jupyter notebooks. Contribute to thetestspecimen/notebooks development by creating an account on GitHub.
thetestspecimen - GitHub

…or get kickstarted in either deepnote or colab.

Python 1.24.0:

Python 1.23.5:

Environment Setup — Local or Deepnote

Whether using a local environment, or deepnote, all that is needed is to ensure that the appropriate version of NumPy is available. The easiest way to achieve this is to add it to your “requirements.txt” file.

For deepnote you can create a file called “requirements.txt” in the files section in the right pane and add the line:

numpy==1.24.0

(change the version number as appropriate).

Environment Setup — Colab

As there is no access to something like a “requirements.txt” file in colab you will need to explicitly install the correct version of NumPy. To do this run the following code in a blank cell to install the appropriate version:

!pip install numpy==1.24.0

(change the version number as appropriate).

Then refresh the web page before trying to run any code.

The test

The following speed tests were run to give an overview of the improvements to the np.in1d and np.equal methods when using the latest NumPy version (1.24.0) compared to the previous version (1.23.5):

Two integer arrays with length 50000 using method np.equal (method run 1 million times) — should be up to 2.75x faster with numpy 1.24.0 according to the documentation
Two boolean arrays with length 50000 using method np.equal (method run 1 million times) — should be up to 19.15x faster with numpy 1.24.0 according to the documentation
Two integer arrays compared using np.in1d using kind="sort" (method run 10 thousand times) — this method is available in both numpy 1.23.5 and 1.24.0 — should be the same speed in numpy 1.23.5 and 1.24.0 (a good cross-check between notebooks to ensure that the results of the other tests can be directly compared)
Two integer arrays compared using np.in1d using the new kind="table" method (method run 10 thousand times) — this method is only available in numpy 1.24.0 — should be up to 10x faster than the “sort” method according to the documentation

Input data

Timeit setup

Results — NumPy 1.23.5

Results-NumPy 1.24.0

Summary

Results comparison — Image by author

As you can see from the results sections, the results achieved are roughly inline with what was expected, and all methods achieve an increase in speed of execution:

Two integer arrays using method np.equal — 4.67x faster with numpy 1.24.0
Two boolean arrays using method np.equal — 15.56x faster with numpy 1.24.0
Two integer arrays compared using np.in1d using the kind="sort" method — more or less exactly the same execution time using numpy 1.23.5 and 1.24.0 as expected (18.5 seconds for 10000 iterations)
Two integer arrays compared using np.in1d using the new kind="table" method — 3.84x faster with numpy 1.24.0 and the newly introduced kind="table" method

Conclusion

The efforts of software developers to constantly improve the languages and libraries that we all use on a daily basis should not be ignored. It is one of the easiest and most accessible ways to improve the efficiency, speed and reliability of your projects code.

As you can see from the very small example outlined in this article, the benefits can be quite significant. All it takes a little organisation, and the willingness to invest some time into reviewing the release notes for your most important libraries / software.

How to Improve Clustering Accuracy with Bayesian Gaussian Mixture Models

Wed, 15 Feb 2023 00:00:00 GMT

In the real world you will often find that data follows a certain probability distribution. Whether it is a Gaussian (or normal) distribution, Weibull distribution, Poisson distribution, exponential distribution etc., will depend on the specific data.

Being aware of which distribution describes your data, or likely best describes your data, allows you to take advantage of that fact, and improve your inference and/or predictions.

This article will look at how leveraging knowledge of an underlying probability distribution of a dataset can improve the fit of a bog standard K-Means clustering model, and even allow for automatic selection of the number of appropriate clusters, directly from the underlying data.

Introduction

A lot of the headline grabbing machine / deep learning techniques tend to involve supervised machine / deep learning i.e. the data has been labelled, and the models are given the correct answers to learn from. The trained model is then applied to future data to make predictions.

This is all very useful, but the reality is that data is constantly being produced by businesses and people around the world, and the majority of it is not labelled. It is actually quite expensive and time consuming to label data in the vast majority of cases. This is where unsupervised learning comes in.

…data is constantly being produced by businesses and people around the world, and the majority of it is not labelled.

Finding the best way to infer meaning from unlabelled data is a very important pursuit for many businesses. It allows the unearthing of potentially unknown, or less obvious, trends or groupings. It is then possible to assign resources, target specific groups of customers, or just instigate additional research and development.

Further to this, a large majority of the time, the data involves people or natural processes in one form or other. Natural processes, and the behaviour of people are, more often than not, captured and described well by a Gaussian distribution.

With this in mind, this article will take a look into how a Gaussian distribution, in the form of both a Gaussian Mixture Model (GMM) and a Bayesian Gaussian Mixture Model (BGMM), can be utilised to improve the clustering accuracy of a dataset that represents ‘natural processes’ encountered in real world datasets.

As a comparison and base for judgement, the ubiquitous K-Means clustering algorithm will be used.

The Plan?

Photo by Christina Morillo from Pexels

This article will cover quite a lot of ground, and also incorporate examples from a comprehensive Jupyter notebook.

This section should give you some guidance on what is covered and where to skip to should you need access to specific information.

Initially the article will cover the “what” and “why” questions in regard to the use of Gaussian Mixture Models in general.
As there are two readily available implementations of Gaussian Mixture Models within the scikit-learn library, a discussion of the key differences between a plain Gaussian Mixture Model and Bayesian Gaussian Mixture Model will follow.
The article will then dive into using Gaussian Mixture Models to cluster a real world multi-featured dataset. All examples will be implemented using K-Means, a plain Gaussian Mixture Model and a Bayesian Gaussian Mixture Model.
There will then be two additional sections primarily focused on clearer visualisation of the algorithms. Complexity of the data will be reduced by a) using a two component principle component analysis (PCA) and b) analysing only two features from the dataset

Note: the dataset chosen is in fact labelled. This has been done deliberately so that the performance of the clustering can be compared to the 100% correct and known primary clustering.

The Explanations

Photo by Polina Zimmerman from Pexels

Here we cover the “what” and “why” type questions before getting stuck into the data to see it all in action.

Before we get started, it is worth noting that there will be no discussion of what a Gaussian / Normal distribution is, or even what K-Means clustering is. There are plenty of great resources out there, and there just isn’t the space to cover it in this article. Therefore, a basic understanding of those concepts is assumed from this point on.

Note: the phrases “Gaussian distribution” and “Normal distribution” mean one and the same thing, and will be used interchangeably throughout this article.

What is a Gaussian Mixture Model?

In simple terms, the algorithm assumes that the data you provide it can be approximated with an unspecified mixture of different Gaussian distributions.

The algorithm will try to extract and separate this mixture of Gaussian distributions and pass them back to you as separate clusters.

That’s it really.

So in essence, it is like K-Means clustering, but it has the added advantage of being able to apply additional statistical constraints on the data. This gives more flexibility to the shape of the clusters it can capture. In addition, it allows closely spaced, or slightly conjoined, clusters to be separated more precisely, as the algorithm has access to the statistical probabilities generated by the assumed Gaussian distributions.

In a visual sense, if the analysis is restricted to two or three dimensions, the K-Means algorithm pins down cluster centres and applies a ‘circular’ or ‘spherical’ distribution around those centres.

However, if the underlying data is Gaussian it is perfectly possible, and even expected, that the distribution will be elongated to some extent due to the tails of a Gaussian distribution. This would be the equivalent of an ‘ellipse’ or ‘ellipsoid’ in terms of shape. These elongated ‘ellipse’ or ‘ellipsoid’ type shapes are something K-Means cannot model accurately, but the Gaussian Mixture Models can.

On the other side of the coin, if you pass the Gaussian Mixture Models data that is definitely nowhere near Gaussian, the algorithm will still assume it is Gaussian. You will therefore likely end up with, at best, something that is no better than K-Means.

Side note: the Gaussian Mixture Model and Bayesian Gaussian Mixture Model can use the K-Means clustering algorithm to generate some of the initial parameters of the model (it is in fact the default setting in scikit-learn).

Which begs the question…

What type of data could Gaussian Mixture Models be used for?

Natural processes are usually a good place to start. The reason for this is mainly due to the Central Limit Theorem (CLT):

In probability theory, the central limit theorem (CLT) establishes that, in many situations, when independent random variables are summed up, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed.

-wikipedia.org

In very simple terms, what this essentially means with natural processes is that although the particular variable (a humans height for example) is caused by many factors that may, or may not, be normally distributed (diet, lifestyle, environment, genes etc.) the normalised sum of those parts (the human height) will be (approximately) normally distributed. That is why we tend to see natural processes appearing to be normally distributed so regularly.

Photo by Ivan Samkov from Pexels

As such, there are a surprisingly large amount of real world instances where it is necessary to deal with Gaussian distributions, making Gaussian Mixture Models a very useful tool if it used in the appropriate setting:

Customer behaviour — this could be in terms of purchases made, amounts spent, attention span, churn etc.
Human characteristics — height, weight, shoe size, IQ (or educational performance) etc.
Natural phenomena — recognition of patterns / groups in the medical field (cancers, diseases, genes etc.) or other scientific fields

There are obviously many other examples, and other situations where a Gaussian distribution may arise for other reasons.

There may of course be cases where you cannot discern the underlying structure of the data, and this will of course warrant investigation, but is also one of the reasons why domain experts can be so important.

Why use a Gaussian Mixture Model?

The main reason is that if you are fairly confident that your data is Gaussian (or more precisely a mixture of Gaussian data) then you will give yourself a much better chance of separating out real clusters with much more accuracy.

The algorithm not only looks for basic clusters, but also considers the most appropriate shape, or distribution, of each cluster. This allows for tightly spaced, or even slightly conjoined, clusters to be separated out more accurately and easily.

Furthermore, where there are cases that the separation is not clear (or maybe just for more in depth analysis) it is possible to produce, and analyse, the probability that each data point belongs to each cluster. This gives you a better understanding of what are core reliable data points, and those that are perhaps marginal, or unclear.

With Bayesian Gaussian Mixture models it is also possible to let the algorithm infer from the data the most appropriate number of clusters. Rather than having to rely on the elbow method or produce BIC / AIC curves.

How does a Gaussian Mixture Model work?

It is basically using an iterative updating process to gradually optimise the fit of a number of Gaussian distributions to the data. I suppose in a similar way to gradient decent.

Check the fit — adjust — check the fit again — adjust…and repeat until converged.

In this case the algorithm is called the expectation-maximisation (EM) algorithm. More specifically this is what happens:

One of the input parameters to the model is the number of clusters, so this is a known quantity. For example, if two clusters are set, then an initial set of two Gaussian distributions will have their parameters assigned. The parameters could be assigned by a K-Means analysis (the default in scikit-learn), or just randomly. The parameters could even be specified specifically for every data point, if you have a very specific case. Now on to the iteration…
Expectation — now there are two Gaussian distributions that are defined with specific parameters. The algorithm first assigns each data point to one of the two Gaussian distributions. It does this based on the probability that it fits into that particular distribution, compared to the other.
Maximisation — once all the points are assigned, the parameters of each Gaussian distribution are adjusted slightly to better fit the data as a whole, based on the information generated from the previous step.
repeat steps 2 and 3 until convergence.

The above is overly simplified, and I haven’t detailed the exact mathematical mechanism (or equations) that are optimised at each point in time. However, it should give you at least a conceptual understanding of how the algorithm operates.

As ever, there are plenty of mathematics heavy articles out there explaining in much more detail the exact mechanisms should that interest you.

What is the difference between a normal Gaussian Mixture Model and a Bayesian Gaussian Mixture Model?

I’m going to start by saying that, to explain the additional processes used by the Bayesian Gaussian Mixture Model over and above the standard Gaussian Mixture Model is actually quite involved. It requires an understanding of a few different, and complicated, mathematical concepts at the same time, and there certainly isn’t space in this article to do it justice.

What I will aim to do here is point you in the right direction, and outline the advantages and disadvantages. You will also gather further information as you pass through the rest of the article while the dataset is analysed.

Practical Differences

The standout difference is that the standard Gaussian Mixture Model uses the expectation-maximisation (EM) algorithm, whereas the Bayesian Gaussian Mixture Model uses variational inference (VI).

Unfortunately, variational inference is not mathematically straight forward, but if you want to get your hands dirty I suggest this excellent article by Jonathan Hui

The main take-aways are these:

variational inference is an extension of the expectation-maximisation algorithm. Both aim to find Gaussian distributions within your data (in this instance at least)
Bayesian Gaussian Mixture Models require more input parameters to be provided, which is potentially more involved / cumbersome
variational inference inherently has a form of regularisation built in
variational inference is less likely to generate ‘unstable’ or ‘marginal’ solutions to the problem. This makes it more likely that the algorithm will tend towards a solidly backed ‘real’ solution. Or as scikit-learn’s documentation puts it “due to the incorporation of prior information, variational solutions have less pathological special cases than expectation-maximization solutions.”
Bayesian Gaussian Mixture Models can directly estimate the most appropriate amount of clusters for the input data (no elbow methods required!)

…so to summarise:

Variational inference is a more advanced extension to the idea behind expectation-maximisation. It should in theory be more accurate and more resistant to messy data or outliers.

Resources and further reading

As a start I would point you to the excellent overview provided in the literature for scikit-learn:

2.1. Gaussian mixture models
sklearn.mixture is a package which enables one to learn Gaussian Mixture Models (diagonal, spherical, tied and full covariance matrices supported), sample them, and estimate them from data. Facilit…
null - scikit-learn

For further reading, some relevant subjects to look up are:

Expectation-Maximisation (EM)
Variational Inference (VI)
The Dirichlet distribution and Dirichlet process

Now for some real data

Photo by Cup of Couple from Pexels

Discussion and theory are great, but I often find exploring a real implementation can clarify a great deal. With that in mind, the following sections will make a comparison of the performance of each of the following clustering methods on a real world dataset:

K-Means (the baseline)
Gaussian Mixture Model
Bayesian Gaussian Mixture Model

The Data

The real world dataset^[1] that will be used describes the chemical composition of three different varieties of wine from the same region of Italy.

This dataset is labelled, so although all the clustering analysis that follows will not use the labels in the analysis, it will allow a comparison to a known correct answer. The three methods of clustering can therefore be compared fairly and without bias.

Furthermore, the dataset meets the criterion of being a “natural” dataset, which should be a good fit for the Gaussian methods that are the intended test target.

To ease the ability to load the data I have made the raw data available in CSV format in my GitHub repository:

notebooks/datasets/wine at main · thetestspecimen/notebooks
Jupyter notebooks. Contribute to thetestspecimen/notebooks development by creating an account on GitHub.
thetestspecimen - GitHub

Reference Notebooks

All the analysis that follows has been made available in a comprehensive Jupyter notebook.

The raw notebook can be found here for your local environment:

notebooks/bayesian_gaussian_mixture_model.ipynb at main · thetestspecimen/notebooks
Jupyter notebooks. Contribute to thetestspecimen/notebooks development by creating an account on GitHub.
thetestspecimen - GitHub

…or get kickstarted in either Deepnote or Colab if you want an online solution:

There are some functions that are used within the notebooks that require certain libraries to be reasonably up to date (specifically scikit-learn and matplotlib), so the following sections will describe what is needed.

Environment Setup — Local or Deepnote

Whether using a local environment, or Deepnote, all that is needed is to ensure that the appropriate version of scikit-learn and matplotlib is available. The easiest way to achieve this is to add it to your “requirements.txt” file.

For Deepnote you can create a file called “requirements.txt” in the files section in the right pane, and add the lines:

scikit-learn==1.2.0
matplotlib==3.6.3

(more recent versions are also ok).

Environment Setup — Colab

As there is no access to something like a “requirements.txt” file in Colab you will need to explicitly install the correct versions of scikit-learn and matplotlib. To do this run the following code in a blank cell to install the appropriate versions:

!pip install scikit-learn==1.2.0
!pip install matplotlib==3.6.3

(more recent versions are also ok).

Then refresh the web page before trying to run any code, so the libraries are properly loaded.

Data Exploration

Before running the actual clustering it might be worth just getting a rough overview of the data.

There are 13 features and 178 examples for each feature (no missing or null data):

…so a nice clean numerical dataset to get started with.

The only thing that really needs changing is the scale. The range of numbers within each of the features varies quite a bit, so a simple MinMaxScaler will be applied to the features so that they all sit between 0 and 1.

Data Distribution

As previously mentioned, it would be ideal if the data was at least approximately Gaussian to allow the Gaussian Mixture Model to work effectively. So how does it look?

Feature data distribution — Image by author

Now some of those look quite Gaussian (e.g. alkalinity of ash), but the reality is most do not. Is this a problem? Well not necessarily, as what really exists in a lot of cases is a *mixture* of Gaussian distributions (or approximate Gaussian distributions).

The whole point of the Gaussian Mixture Model is that it can find and separate out the individual Gaussian distributions from a mixture of more than one Gaussian distribution.

In a real clustering problem you would not be able to achieve this next step, as you wouldn’t know the real clustering of the data. However, just for illustration purposes it is possible (as we know the labels) to plot each individual ‘real’ clusters distribution:

Feature data distribution by label — Image by author

As you can now see the real clusters are in most cases approximately Gaussian. This is not the case across the board, but it will never be the case with real data. As expected, due to the data being based on “natural” data we are indeed dealing with approximately Gaussian data.

This very basic investigation of the distribution of the raw data illustrates the importance of being aware of the type of data you are dealing with, and which tools would be best suited to the analysis. This is also a good case for the importance of domain experts, where appropriate.

Feature relations

Finally, a few examples of how some of the variables are distributed in relation to each other (if you want a more comprehensive plot please take a look at the Jupyter notebook):

An example of the distribution of data between components — Image by author

As can be seen from the scatter plots above, some features show reasonable separation. However, there is also quite a lot of mixing with some features, and more so on the periphery of each cluster. The shape (circular, elongated etc.) also varies quite widely.

The Analysis

Photo by Artem Podrez from Pexels

As mentioned earlier, there will be three different algorithms compared:

K-Means (the baseline)
Gaussian Mixture Model
Bayesian Gaussian Mixture Model

There will also be three phases to the exploration:

All of the raw data analysed at once
All of the data analysed after reducing complexity / features with a Principle Component Analysis (PCA)
An analysis using just two features (mainly to allow easier illustration than using the full dataset)

Analysis 1 — the full dataset

To keep the comparison consistent, and reduce the complexity of this article I am going to skip a thorough investigation into how many components is the correct number.

However, just for completeness you should be aware that a crucial step in performing any clustering is gaining an understanding of the appropriate amount of clusters to use. Some common examples of how to achieve this are the elbow method, the Bayesian Information Criterion (BIC) and the Akaike information criterion. As an example here are the BIC and AIC for the whole raw dataset:

The BIC and AIC curves for this dataset — Image by Author

The BIC would suggest two components is appropriate, but I’m not going to go further into this result for now. However, it will come up in discussion later in the article.

Three clusters will be assumed from now on for consistency and ease of comparison.

Let’s get stuck in!

K-Means

K-Means confustion matrix — Image by author

Well that is a fairly impressive result.

With all data taken together there is apparently very good separation between clusters 1 and 3. However, as cluster 2 generally sits between clusters 1 and 3 (refer back to the four example scatter plots in the previous section) it would appear that the points on the periphery of cluster 2 are being wrongly assigned to clusters 1 and 3 (i.e. the boundaries between those clusters are likely incorrectly defined).

Let’s see if this improves with a Gaussian Mixture Model.

Gaussian Mixture Model

Gaussian Mixture Model confusion matrix — Image by author

An improvement. The Gaussian Mixture Model has managed to pick up three extra points and assign them correctly to cluster 2.

Although that doesn’t seem like a big deal, it is worth bearing in mind that the dataset is quite small, and using a more comprehensive dataset would likely yield a more impressive number of points.

Bayesian Gaussian Mixture Model

Before producing the result, and as this is the main focus of the article, I think it is worth taking a little time to explain some of the relevant parameters that can be specified:

n_components — this is the number of clusters that you want the algorithm to consider. However, the algorithm may return, or prefer, less clusters than set here, which is one of the main advantages of this algorithm (we will see this in action soon)
covariance_type — there are four options here full, tied, diag and spherical. The most ‘accurate’ and typically preferred is full. This parameter essentially decides the limitation of the distribution fit shape, a great illustration is provided here.
weight_concentration_prior_type — this can either be dirichlet_process (infinite mixture model) or dirichlet_distribution (finite mixture model). In general, it is better to opt for the Dirichlet process as it is less sensitive to parameter changes, and does not tend to divide natural clusters into unnecessary sub-components as the Dirichlet distribution can sometimes do.
weight_concentration_prior — specifying a low value (e.g. 0.01) will cause the model to set a larger number of components to zero leaving just a few components remaining with significant value. High values (e.g. 100000) will tend to allow a larger number of components to remain active with relevant values i.e. less components will be set to zero.

As there are quite a lot of extra parameters it may be wise to perform cross validation analysis in some cases. For example, in this initial run covariance_type will be set to ‘diag’ rather than ‘full’ as the suggested cluster number is more convincing. I suspect in this specific case this is due to a combination of a smaller dataset, and a large number of features causing the ‘full’ covariance type to over-fit.

It is now possible to review what the algorithm has decided in terms of relevant clusters. This is possible by extracting the weights from the fitted model:

A bit more clearly in graph form:

Bayesian Gaussian Mixture Model group weights — Image by author

To be honest, not very convincing.

It is clear that there are three clusters ahead of the rest, but this is only intuitive because the answer is known. The truth is it is not very clear. So why is this?

The main reason is likely due to a combination of lack of data, and a high number of features (at least compared to the amount of data).

Lack of data will cause items such as outliers to have a much larger effect on the overall distribution, but also reduce the models ability to generate a distribution in the first place.

In the sections that follow we will look at a way to potentially combat this, and also at analysing a smaller selection of features to see what result we get in those circumstances.

For now, as we know the correct number of components is three we will push on.

Bayesian Gaussian Mixture Model confusion matrix — Image by author

Almost perfect clustering. Only two points remain incorrectly assigned.

Let’s take a quick look at an overall comparison.

Accuracy comparison of the different clustering methods — Image by author

Discussion

As can be seen from the accuracy results, there is a gradual improvement from one method to the next. This goes to show that understanding the structure of the underlying data is important, as it allows a more accurate representation of the patterns within the raw data, in our case Gaussian distributions.

Even within the realm of Gaussian Mixture Models, it is also clear (at least in this case) that the use of variational inference in the Bayesian Gaussian Mixture Model can yield more accurate results than expectation-maximisation.

All of this was expected, but it is interesting to see nonetheless.

Visualisation

To give a general overview of what exactly the Gaussian algorithms are doing, I have plotted the feature “alcohol” against each of the other features, and also included the confidence ellipses of the Bayesian Gaussian Mixture Model on each plot for each of the three clusters.

Note: the algorithm used to draw the confidence ellipses is adapted from this algorithm provided in the scikit-learn documentation.

Bayesian Gaussian Mixture Model - Alcohol compared to all other features — Image by author

Although this is interesting to look at, and does do a good job of showing how the algorithm can account for the various shapes of the clusters (i.e. round, of more elongated ellipses) it doesn’t allow any sort of understanding as to what factors may have influenced the outcome.

This is mainly due to the fact that there are too many dimensions to deal with (i.e. too many features) in terms of representing the output as something interpretable.

Better visualisation and further investigation

It would be interesting to visualise and compare the distributions produced by the various methods in a clear and concise way, so that it is possible to see exactly what is going on under the hood.

However, due to the large amount of features (and therefore dimensions) the model is processing, it is not possible to represent what is going on in a simple 2D graph, as has just been illustrated in the previous section.

With this in mind the aim of the next two sections of this article is to simplify the analysis by reducing the dimensions. The following two sections of this article will therefore look into:

the use of a two component Principle Component Analysis (PCA). This will allow all of the thirteen features to be distilled into two features, whilst keeping the *majority* of the important information embedded in the data.
specifically using only two features to run the analysis.

Both of these investigations will allow a direct visualisation of the clusters as there are only two components, and therefore it is then possible to plot them on a standard 2D graph.

Furthermore, these approaches will potentially make it easier to use the Bayesian Gaussian Mixture Model’s automatic cluster selection more effectively, as the amount of features in relation to the number of examples is more in balance.

Analysis 2 — Principle Component Analysis (PCA)

By running a two component Principle Component Analysis (PCA) the 13 features can be compressed down to 2 components.

The main aims for this section are as follows:

Gain insights into the data distribution from the reduced complexity afforded by the PCA before analysis.
Review the automatic component selection generated by the Baysian Gaussian Mixture Model from the PCA dataset.
Run the Bayesian Gaussian Mixture Model on the two PCA components, and review the clustering result in 2D graph form.

The result of the PCA

The two components of the PCA on all the data with distributions (colours are real label clusters) — Image by Author

The result of the PCA is interesting for quite a few reasons.

Firstly, the distribution of the primary PCA component (and to some degree the secondary PCA component) is very close to a Gaussian distribution for all three components. Mirroring what we have discovered from the data investigation earlier in the article.

Remembering for a second that the PCA is a distillation of all of the features, it is interesting to see that there is good separation between the three real clusters. This means there is a very good possibility that a clustering algorithm that targets the data well (regardless of the PCA) has the potential to achieve a good and accurate separation of the clusters.

However, the clusters are close enough, that if the clustering algorithm does not represent the distribution, or shape, of the clusters correctly, there is no guarantee that the boundary between the clusters will be found accurately.

For example, the distribution of cluster 1 is quite clearly elongated along the y-axis. Any algorithm would need to mimic this elongated shape to represent that particular cluster correctly.

Automatic Clusters

In the initial analysis the automatic estimation of the correct number of clusters was a little ambiguous. Now that the data complexity has been reduced let’s see if there is any improvement.

Again, the inputs request 8 components:

The weights of each of the 8 requested clusters — Image by Author

…and there we are. A very clear indication that even though the model was set up to consider up to 8 clusters, the algorithm clearly thinks that the actual appropriate number of clusters is 3. Which we of course know to be correct.

This goes some way to indicate that for the Bayesian Gaussian Mixture Model to work effectively, in terms of automatic cluster selection, it is necessary to consider whether the dataset is large enough to generate a reasonable and definitive result if the number of clusters is not already known.

The result

For completeness let’s see how the clustering for the Bayesian Gaussian Mixture Model of the PCA went.

Bayesian Gaussian Mixture Model (PCA) confusion matrix — Image by author

A definite accuracy drop when compared to using the raw data. This is of course expected. By running a PCA we are definitely losing data, there is no avoiding that, and in this case it is enough to introduce an additional 5 misassigned data points.

Let’s look at this visually in a little more detail.

Bayesian Gaussian Mixing of a two component PCA (mismatched points ringed red) — Mismatched points from original analysis of all data ringed in blue — Image by Author

The graph above has a lot going on, so let’s break this down:

the coloured dots are the clusters that the PCA data was assigned to by the algorithm.
the large shaded ellipses are the confidence ellipses, which essentially state the shape of the underlying distribution generated by the algorithm (co-variances).
the red circles are the data points that have been misassigned by the Bayesian Gaussian Mixing Model from the PCA analysis.
the blue circles are the data points that were misassigned by the Bayesian Gaussian Mixing Model in the original analysis that used all of the raw data.

What is particularly interesting is that by running the PCA analysis, and likely due to the inherent loss of data, it forces some data points to be shifted well within the clusters / confidence ellipses that are generated (look at the orange point labelled with text, and dot circled blue within the green confidence ellipsoid).

In the case of the blue circled green point it helped, it is an improvement over the original ‘all raw data’ analysis. However, it is quite clear the orange point that is misassigned would never be correctly assigned, as it too well embedded within the orange cluster, when in fact it should be in the green cluster. However, the original analysis correctly assigned this data point to the green cluster.

PCA Summary

In this particular case, it was useful with a smaller dataset to run a PCA to help the Bayesian Gaussian Mixture Model fix on an appropriate number of clusters. Even as an aid in getting a good visual grasp of how the dataset is distributed, it is potentially very useful.

However, it would not be an optimal solution to use the data generated by the PCA as a final input into the analysis. It is clear that the data shift / loss is sufficient enough as to potentially make some data points permanently wrongly assigned, and the overall accuracy is also reduced.

Analysis 3 — Fewer features

To take a closer look at the differences between the three clustering algorithms two specific features will be extracted, and the clustering algorithms run on only those features.

The advantage of this in terms of reviewing the methods, is that it becomes possible to visualise what is going on a 2D-plane (i.e. a normal 2D graph).

An added bonus, is that the available data has much less information than the whole dataset. This both reduces the discrepancy between the small number of samples and the number of features, whilst also forcing the clustering algorithms to work much harder to achieve an appropriate fit due to the reduced information.

A closer look at the data

Colour intensity vs OD280/OD315 of diluted wines (raw data with labels) — Image by author

The reason for picking this pair (colour intensity and OD280/OD315 of diluted wines) is due to the challenges the dataset throws up for the different clustering algorithms.

As you can see there are two clusters that are fairly intermingled (1 & 2). In addition, clusters 2 & 3 have quite elongated distributions compared to cluster 1, which is more rounded, or circular.

In theory, the Gaussian mixing methods should fair quite a bit better than K-Means as they have the ability to accurately mould their distribution characteristics to the elongated distributions, whereas K-Means cannot, as it is limited to a circular representation.

Furthermore, from the KDE plots at each side of the graph it is possible to see that the data distribution is reasonably Gaussian as we have confirmed a few times before in this article.

K-Means — Results

Confusion matrix for the K-Means - Reduced features — Image by author

Gaussian Mixture Model — Results

Confusion matrix for the Gaussian Mixture Model — Reduced features — Image by author

Bayesian Gaussian Mixture Model — Component selection

Before diving straight into the results, as the Bayesian Gaussian Mixture Model has the ability to auto-select the appropriate number of components we will again ‘request’ 8 components and see what the model suggests.

Component selection for the Bayesian Gaussian Mixture Model with two features only — Image by author

As previously seen with the reduced complexity of the PCA analysis, the model has found it much easier to distinguish the 3 clusters that are known to exist.

This would fairly conclusively confirm that:

it is necessary to have sufficient data samples to ensure that the model can stand a decent chance of offering the correct suggestion for number of clusters. This is likely due to the need to have enough data to properly represent the underlying distribution. In this case Gaussian.
a potential way to combat lack of samples is to in some way simplify or generalise the data to try and extract the appropriate number of clusters, and then revert to the full dataset for a final full clustering analysis

Bayesian Gaussian Mixture Model — Results

Confusion matrix for the Bayesian Gaussian Mixture Model — Reduced features — Image by author

Final results comparison

The accuracy of each clustering method for features colour intensity and OD280/OD315 of diluted wines — Image by author

As expected, the accuracy is a lot lower than when using the full dataset. However, regardless of this fact, there are some stark differences between the accuracy of the various methods.

Let’s dig a little deeper by comparing everything side-by-side.

A comparison of the assignment of clusters for each of the three clustering methods — Cluster 1 (red) / Cluster 2 (orange/green) / Cluster 3 (grey/blue) — Image by author

The first thing to note is that the real labels show that there is a certain amount of fairly deep mixing / crossover between the clusters in some instances, so 100% accuracy is out of the question.

Reference for the following discussion:

Cluster 1 — Red
Cluster 2 — Orange/Green
Cluster 3 — Grey/Blue

K-means

K-means does a reasonable job of splitting out the clusters, but due to the fact that the distributions must ultimately be limited to a circular shape, there was never any hope of accurately capturing clusters 2 or 3 precisely.

However, as cluster 3 is quite well separated from the other two, the elongated shape is less of a hindrance. A circular representation of the lower cluster is actually sufficient, and gives a comparable representation to the Gaussian methods.

When considering clusters 1 and 2 the K-Means method fails to sufficiently represent the data. It has an inherent inability to properly represent the elliptical shape of cluster 2. This causes cluster 2 to be ‘squashed’ down in between clusters 1 and 3 as the real extension upwards cannot be sufficiently described by the K-Mean algorithm.

Gaussian Mixture Model

The basic Gaussian Mixture Model is only a slight improvement in this case.

As discussed in the previous section for K-Means, even though the distribution of cluster 3 is better suited to an elliptical rather than circular distribution, it is of no advantage in this particular scenario.

However, when looking at the representation of cluster 1 and 2, which are significantly more intertwined, the ability of the Gaussian Mixture Model to represent the underlying elliptical distribution (i.e. better capturing the underlying tails of the Gaussian distribution) of cluster 2 more accurately, results in a slight increase in accuracy.

Bayesian Gaussian Mixture Model

For a start, the more than 10% improvement in accuracy of the Bayesian Gaussian Mixture Model compared to the other methods is certainly impressive as a lone statistic.

On review of the confidence ellipsoids it becomes clear why this is the case. Cluster 2 has been represented both in terms of shape, tilt and size just about as perfectly as it could be. This has allowed for a very accurate representation of the real clusters.

Although the exact reasons for this will always be slightly opaque, it is most definitely down to the differences between the expectation-maximisation algorithm used by the standard Gaussian Mixture Model, and the variational inference used by the Bayesian Gaussian Mixture Model.

As discussed earlier in the article, the main differences are:

the in built regularisation
less tendency for variational inference to generate ‘marginally correct’ solutions to the problem

Conclusion

It is quite clear that the use of Gaussian Mixture Models can help to elevate the accuracy of the clustering of data that is likely to have Gaussian distributions as it’s underpinning.

This is particularly relevant and useful for natural processes, including human processes, which make this analytical approach relevant to a large number of industries, across a wide variety of fields.

Furthermore, the introduction of variational inference in the Bayesian Gaussian Mixture Model can, with very little difference in overhead, return further improved accuracy in clustering. There is even the, not so insignificant bonus, of the algorithm having the ability to suggest the appropriate amount of clusters for the underlying data.

I hope this article has provided you with a decent insight into what Gaussian Mixing Models and Bayesian Gaussian Mixture Models are, and whether they may help with data you are working with.

They really are a powerful tool if used appropriately.

References

[1] Riccardo Leardi, Wine (1991), UC Irvine Machine Learning Repository, License: CC BY 4.0

Pro GPU System vs Consumer GPU System for Deep Learning

Wed, 19 Apr 2023 01:00:00 GMT

Having a GPU (or graphics card) in your system is almost essential when it comes to training neural networks, especially deep neural networks. The difference in training speed of a fairly modest GPU is a night and day difference when compared to a CPU.

….but at what point might you consider jumping into the realms of professional, rather than consumer, level GPUs? Is there a huge difference in training and inference speed? Or is it other factors that make the jump compelling?

Introduction

The aim of this article is to give you an idea of the main differences between a GPU you might use as a normal consumer (or starting out in machine/deep learning), and those used in higher end systems. The type of systems that might be used in the development and/or inference of advanced deep learning models.

Apart from being an interesting exercise in understanding the distinctions between cutting edge pro equipment and consumer level hardware in terms of pure processing speed, it will also highlight some of the other limitations that are present in consumer level GPUs, and associated systems, when dealing with cutting edge deep learning models.

Which GPUs are you referring to when you say “Pro” or “Consumer”?

The “real world” differences will be covered in the rest of the article, but if you want a solid technical distinction, with example graphics cards and specifications, then this section should cover it.

As detailed in one of my previous articles, NVIDIA are the only sensible option when it comes to GPUs for deep learning and neural networks at the current time. This is mainly due to their more thorough integration into platforms such as TensorFlow and PyTorch.

Making the distinction between professional and consumer, in terms of specifications from the manufacturer, is therefore relatively straight forward.

Anything from the following page is the current batch of NVIDIA’s consumer graphics cards:

NVIDIA GeForce Graphics Cards
GeForce RTX 50 series, 40 series & more.
null - NVIDIA

…and professional level GPUs:

NVIDIA RTX PRO in Desktops
The World’s Most Powerful Platform for AI, Graphics, and Simulation.
null - NVIDIA

There are also GPUs, mainly used in data centres, that go beyond even the pro level graphics cards listed above. The A100 is a good example:

NVIDIA A100 GPUs Power the Modern Data Center
The fastest data center platform for AI and HPC.
null - NVIDIA

You can also get an idea of the system specifications and GPUs being used in professional data centre and workstation systems certified by NVIDIA:

NVIDIA-Certified Systems — NVIDIA Certification Programs Documentation
This document lists the systems that have been tested with the latest NVIDIA GPUs and networking and are evaluated by NVIDIA engineers for performance, functionality, scalability, and security.
null - nvidia.com

The Plan

I generally find that a real demonstration (or experiment) is the best way to illustrate a point, rather than just relying on specs and statistics provided by the manufacturers.

With that in mind, although the article will discuss the relevant statistics, it will also directly compare three different GPUs (pro and consumer), at differing levels of sophistication, on the same deep learning model.

This should help to highlight what is important, and what isn’t, when considering whether a professional level GPU is for you.

The GPU Specs — A Summary

Photo by Ann H on Pexels

For the experiment, there will be three different graphics cards, but four levels of comparison:

NVIDIA RTX 1070 (Basic)
NVIDIA Tesla T4 (Mid range)
NVIDIA RTX 6000 Ada (High end)
2 x NVIDIA RTX 6000 Ada (Double high end!)

So how do these different graphics cards compare in terms of raw specs?

GPU	Memory	CUDA Cores	Tensor Cores	FP32 [TFLOPS]	Generation	Power
RTX 6000 Ada	48GB	18176	568	91.1	8.9 (Ada Lovelace)	300W
RTX 4090	24GB	16384	512	82.6	8.9 (Ada Lovelace)	450W
Tesla T4	16GB	2560	320	8.14	7.5 (Turing)	70W
GTX 1070	8GB	1920	0	6.46	6.1 (Pascal)	150W

Comparison of different graphics cards

Note: I have included the RTX 4090 in the table above as it is the pinnacle of current consumer level graphics cards, and probably the best direct comparison to the RTX 6000 Ada. I will reference the 4090 throughout the article as a comparison point, although it will not feature in the benchmarks.

If the table above is just a load of number with no meaning, then I recommend my previous article which goes over some of the jargon:

How to Pick the Best Graphics Card for Machine Learning | Towards Data Science
Speed up your training, and iterate faster
Michael Clayton - Towards Data Science

The Professional System

A view from all angles of the professional workstation utilised in this article. Image via Exxact Corporation under license to Michael Clayton

One of the problems with producing an article like this is that you need access to a professional level system, and therefore one of the main hurdles is…cost.

Fortunately, there are companies out there that will give access to their equipment for trial runs, to allow you to see if it fits your needs. In this particular case Exxact have been kind enough to allow remote access to one of their builds for a limited period so I can get the comparisons I need.

…the workstation is worth in the region of USD 25,000

To drive home my point about how much these systems can cost I estimate the workstation I have been given access to is worth in the region of USD 25,000. If you want to depress (or impress?) yourself further you can take a look at the configurator and see what can realistically be achieved.

Incidentally, if you are seriously in the market for this level of hardware you can apply for a remote ‘‘test drive’’ too:

ExxactAccess+ Test Drive | Validate Workloads on 8x GPUs | Exxact Corp
ExxactAccess+ offers remote validation with up to 8x NVIDIA RTX PRO Blackwell GPUs. Validate your unique workload performance before you invest in your new Exxact system.
null - exxactcorp.com

These are the complete specs of the “professional” system for those that are interested:

Component Type	Component
Motherboard	Asus Pro WS WRX80E-SAGE SE WIFI-SI
CPU	AMD Ryzen Threadripper PRO 5995WX 64-core 2.7GHz/4.5GHz Boost 280W
GPU	2x NVIDIA RTX 6000 Ada
RAM	8x 64GB DDR4 3200MHz ECC Reg
SSD	1TB M.2 NVMe PCIe 4.0 / 4TB M.2 NVMe PCIe 4.0
Power	2000W Modular ATX PS2 80Plus Platinum
Cooling	360mm AIO CPU Liquid Coole / 3x 120mm 1700RPM Fans
Case	Fractal Meshify 2 XL

Specification of the professional system

Note: Feel free to refer to any of the images in this article that included the two gold looking GPUs in a black computer case, as those are actual pictures of the system detailed above.

It is interesting to note that having a high end system is not just about stuffing the best graphics card you can get your hands on into your current system. Other components need to scale up too. System RAM, motherboard, CPU, cooling, and of course power.

The Contenders

The NVIDIA GeForce GTX 1070 FTW that will be used as the base in this article

NVIDIA GeForce GTX 1070

At the bottom of the pack is the GTX 1070, which is readily available to most people, but is still significantly faster than a CPU. It also has a decent amount of GPU RAM at 8GB. A good simple consumer level base.

NVIDIA Tesla T4

The Tesla T4 is maybe a strange addition, but there are a few reasons for this.

The first thing to note is that the Tesla T4 is actually a professional graphics card, it is just a few generations old.

In terms of processing speed it is roughly the equivalent of an RTX 2070, but it has double the GPU RAM at 16GB. This additional RAM puts it firmly in the mid range of this test. Current generation consumer cards tend to have RAM in this range (RTX 4070 [12GB] and RTX 4080 [16GB]), so it represents consumer graphics cards in terms of GPU RAM quite nicely.

The final reason is that you can easily access one of these graphics cards for free in Colab. That means anybody reading this article can get their hands dirty and run the code to see for themselves!

The Pro GPU — NVIDIA RTX 6000 Ada

A view of the two NVIDIA RTX 6000 Ada graphics cards installed in the professional workstation utilised in this article. Image via Exxact Corporation under license to Michael Clayton

There is no doubt about it, the RTX 6000 Ada is an impressive graphics card both in terms of specs…and price. With an MSRP of USD 6,800 it is definitely not a cheap graphics card. So why would you buy one (or more!?) if you can get an RTX 4090 for a mere USD 1599 (MSRP)?

The RTX 4090 has half the RAM and uses 50% more power than the RTX 6000 Ada

I slipped an RTX 4090 into the table to attempt to answer this question. It helps to demonstrate what tend to be the two most obvious differences between a consumer graphics card and a professional graphics card (at least from the specifications alone):

the amount of GPU RAM available
the maximum power draw in use

The RTX 4090 has half the RAM and uses 50% more power than the RTX 6000 Ada. This is no accident, as will become evident as the article progresses.

Furthermore, considering the higher power draw of the RTX 4090, it is also worth noting that the RTX 6000 Ada is still roughly 10% faster.

Does this additional RAM and reduced power consumption really make a difference? Hopefully, the comparison will help to answer that later in the article.

Any other less obvious advantages?

Well, yes. There are a few additional benefits to getting a professional level graphics card.

Reliability

Image by WikiImages from Pixabay

NVIDIA RTX professional graphics cards are certified with a broad range of professional applications, tested by leading independent software vendors (ISVs) and workstation manufacturers, and backed by a global team of support specialists.

-nvidia.com

In essence this means the graphics cards are likely to be more reliable and crash resistant, both on a software (drivers), and hardware level, and if you do have a problem, there is an extensive professional network available to solve the problem. These factors are obviously very important for enterprise applications where time is money.

Imagine running a complicated deep learning model for a few days and then losing the results due to a crash or bug. Then spending a significant amount more time potentially dealing with the problem. Not good!

Is this peace of mind an additional reason to pay up? That really depends on your priorities, and scale...

Scale

Photo by Daniele Levis Pelusi on Unsplash

If you are designing a computer system to have optimal GPU processing power, then it may well be that you need more than one GPU. There will obviously be a limit on how many GPUs can fit in the system based primarily on the availability of motherboard slots, and physical space constraints in the case.

However, there are other limiting factors directly related to the GPU itself, and this is where consumer GPUs and professional GPUs start to deviate in terms of design.

Consider the fact that a professional motherboard may have availability for four dual slot GPUs (like the pro system in this article). So in theory you could fit 4 x RTX 6000 Ada GPUs into the system no problem at all. However, you would only be able to fit 2 x RTX 4090 on the same board. Why? Because the 4090 is a triple slot graphics card (~61mm thick), whereas the 6000 is a dual slot graphics card (~40mm thick).

Consumer level GPUs are just not designed with the same constraints in mind (i.e. high density builds), and therefore start to be less useful as you scale up.

Cooling

Following on from the potential sizing problem…even if the consumer graphics card was the same dual slot design, there are further issues.

Pro level GPUs tend to be built with cooling systems (blower type) that are designed to draw air through the graphics cards from front to back with a sealed shroud to direct the air straight out of the case (i.e. no hot air recirculating through the case). This allows for pro GPUs to be stacked tightly into the case, and still be able to efficiently cool themselves. All with minimal impact on other components, or GPUs, in the rest of the case.

Two graphics cards one utilising a ‘blower’ cooling system and the other a more common fan cooling system. Photo by Nana Dua on Pexels. Annotations by author.

Consumer GPUs, on the whole, tend to use fan cooling from above/below. This inevitably means hot air from the GPU will recirculate in the case to some degree, necessitating excellent case ventilation.

However, in cases with multiple GPUs, the close proximity of the other graphics cards would make fan cooling very ineffective, and will inevitably lead to sub-optimal temperatures for both the GPUs, and other components in close proximity.

All-in-all professional graphics cards are designed to be tightly and efficiently packed into systems, whilst also staying cool and self contained.

Accuracy

Photo by Ricardo Arce on Unsplash

This really isn’t particularly relevant to deep learning specifically, but pro GPUs tend to have ECC (Error-Correcting Code) RAM. This would be useful where high precision (i.e. a low level of potential random errors from bit flips) is a must for whatever processes you are running through the graphics card.

However, deep learning models are sometimes tuned to be less numerically precise (half-precision 8-bit calculations), so this is not something that is likely to be of real concern for the calculations being run.

Although if those random bit flips happen to crash your model, then it may just be worth consideration too.

The Deep Learning Model

Photo by Pixabay

For the deep learning model I wanted something that is advanced, industry leading, and demanding for the GPUs. It also has to be scalable in terms of difficulty as the GPUs on test have a wide range of capabilities.

A pro level model, for a pro level graphics card

For the model to be industry standard rules out building a model from scratch, so for this comparison an existing, tried and tested, model will be used by utilising transfer learning.

Heavy data

To ensure the input data is heavy, the analysis will be image based, specifically image classification.

Scalability

The final criteria is scalability, and there is a particular set of models out there that fits this criteria perfectly…

EfficientNet

EfficientNet consists of a family of image classification models (B0 to B7). Each model gets progressively more complicated (and accurate). It also has a different expected input shape for the images that you feed in as you progress through the family of models, which increases data input size.

EfficientNet	Parameters	FLOPs	ImageNet Accuracy	Input Shape
B0	5.3M	0.39B	77.1%	224x224x3
B1	7.8M	0.70B	79.1%	240x240x3
B2	9.2M	1.0B	80.1%	260x260x3
B3	12M	1.8B	81.6%	300x300x3
B4	19M	4.2B	82.9%	380x380x3
B5	30M	9.9B	83.6%	456x456x3
B6	43M	19B	84.0%	528x528x3
B7	66M	137B	84.3%	600x600x3

Comparison of the different EfficientNet models — Data from EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks — Table by author

This has a two fold effect:

as you progress through the different EfficientNet models, the model parameters will increase (i.e. a more complicated and demanding model for the GPUs to process)
the volume of raw data that needs to be processed will also increase (ranging from 224x224 pixels up to 600x600 pixels)

Ultimately, this gives a large range of possibilities in terms of loading the GPUs both in terms of processing speed and GPU RAM requirements.

The Data

The data¹ utilised in this article is a set of images which depict the three possible combinations of hand position used in the game rock-paper-scissors.

Four examples from the three different categories of the dataset. Composite image by Author.

Each image is of type PNG, and of dimensions 300(W) pixels x 200(H) pixels, in full colour.

The original dataset contains 2188 images in total, but for this article a smaller selection has been used, which comprises of precisely 2136 images (712 images for each category). This slight reduction in total images from the original has been done simply to balance the classes.

The balanced dataset that was used in this article is available here:

The Test

Photo by Kelly Sikkema on Unsplash

As mentioned previously, there are various levels of EfficientNet available, so for the purposes of testing, the following will be run on each GPU:

EfficientNet B0 (simple)
EfficientNet B3 (intermediate)
EfficientNet B7 (intensive)

This will test the graphics cards speed capabilities due to the difference in overall parameters of each model, but also a wide range of RAM requirements as the input image sizes will vary too.

The EfficientNet models will have all of their layers unlocked and allowed to learn.

The three final models:

EfficientNetB0
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_layer (InputLayer)    [(None, 224, 224, 3)]     0         
                                                                 
 data_augmentation (Sequenti  (None, 224, 224, 3)      0         
 al)                                                             
                                                                 
 efficientnetb0 (Functional)  (None, None, None, 1280)  4049571  
                                                                 
 global_avg_pool_layer (Glob  (None, 1280)             0         
 alAveragePooling2D)                                             
                                                                 
 output_layer (Dense)        (None, 3)                 3843      
                                                                 
=================================================================
Total params: 4,053,414
Trainable params: 4,011,391
Non-trainable params: 42,023
_________________________________________________________________


EfficientNetB3
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_layer (InputLayer)    [(None, 300, 300, 3)]     0         
                                                                 
 data_augmentation (Sequenti  (None, 300, 300, 3)      0         
 al)                                                             
                                                                 
 efficientnetb3 (Functional)  (None, None, None, 1536)  10783535 
                                                                 
 global_avg_pool_layer (Glob  (None, 1536)             0         
 alAveragePooling2D)                                             
                                                                 
 output_layer (Dense)        (None, 3)                 4611      
                                                                 
=================================================================
Total params: 10,788,146
Trainable params: 10,700,843
Non-trainable params: 87,303
_________________________________________________________________


EfficientNetB7
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_layer (InputLayer)    [(None, 600, 600, 3)]     0         
                                                                 
 data_augmentation (Sequenti  (None, 600, 600, 3)      0         
 al)                                                             
                                                                 
 efficientnetb7 (Functional)  (None, None, None, 2560)  64097687 
                                                                 
 global_avg_pool_layer (Glob  (None, 2560)             0         
 alAveragePooling2D)                                             
                                                                 
 output_layer (Dense)        (None, 3)                 7683      
                                                                 
=================================================================
Total params: 64,105,370
Trainable params: 63,794,643
Non-trainable params: 310,727
_________________________________________________________________

Speed Test

The speed of the GPUs will be judged by how quickly they can complete an epoch.

To be more specific, there will be a minimum of two epochs run on each graphics card, and the second epoch will be used to judge the processing speed. The first epoch generally has some additional loading time, so would not be a good reference for general execution time.

The time to run the first epoch will be listed only for reference.

GPU RAM Test

To test the limits of the GPU RAM, the batch size for each graphics card, and each EfficientNet model (i.e. B0, B3 or B7), has been tuned to be as close as possible to the limit for that particular graphics card (i.e. to fill the GPU RAM as much as possible).

The actual peak GPU RAM utilisation for the run will also be disclosed for comparison.

The Code

Photo by Oskar Yildiz on Unsplash

As ever, I have made all the python scripts (GTX 1070 and RTX 6000 Ada) and notebooks (Tesla T4) available on GitHub:

notebooks/pro-vs-consumer-graphics-card at main · thetestspecimen/notebooks
Jupyter notebooks. Contribute to thetestspecimen/notebooks development by creating an account on GitHub.
thetestspecimen - GitHub

You can also access the notebooks for the Tesla T4 directly on Colab if you so wish:

EfficientNetB0:

EfficientNetB3:

EfficientNetB7:

The Results

Photo by Pixabay

EfficientNet B0

Card	1st Epoch [s]	2nd Epoch [s]	Batch Size	Peak RAM [GB]
GTX 1070	47 (+30s)	17	60	6.91
Tesla T4	67 (+40s)	17	128	13.2
RTX 6000 ada	26 (+20s)	4	512	47.7
2 x RTX 6000 ada	39 (+38s)	1	1024	47.7 (each)

EfficientNet B3

Card	1st Epoch [s]	2nd Epoch [s]	Batch Size	Peak RAM [GB]
GTX 1070	108 (+46s)	62	16	5.9
Tesla T4	129 (+74s)	55	40	13.7
RTX 6000 ada	46 (+33s)	13	128	40.6
2 x RTX 6000 ada	65 (+57s)	8	256	41.2 (each)

EfficientNet B7

Card	1st Epoch [s]	2nd Epoch [s]	Batch Size	Peak RAM [GB]
GTX 1070	953 (+95s)	858	1	5.1
Tesla T4	839 (+161s)	678	2	9.9
RTX 6000 ada	213 (+69s)	144	10	44.6
2 x RTX 6000 ada	203 (+125s)	78	20	41.8 (each)

Note: for the first epoch I have listed a number of seconds in brackets. This is the time difference between the first and second epoch.

Discussion — Execution Speed

Image by Arek Socha from Pixabay

The first item to look at is execution speed.

EfficientNet B0 doesn’t cause much of a challenge for any of the graphics cards with this particular dataset, with all completing an epoch in a matter of seconds.

However, it is important to remember that the dataset utilised in this article is small, and in reality the two RTX 6000 Ada graphics cards are approximately 17 times faster that the GTX 1070 (and Tesla T4) in terms of execution speed. The story is pretty much the same for EfficientNet B3 (8x faster) and B7 (11x faster).

The difference is that this slow down in speed, when viewed as execution time, starts to become more of a hindrance the more complicated the model gets.

For example, to execute a single epoch, on this very small dataset, using EfficientNet B7 with a GTX 1070 takes approximately 15 mins. Compare that to just over 1 minute with a pair of RTX 6000 Ada.

…and it gets worse.

Scaling up

Let’s be realistic. No model is going to converge in one epoch. Four hundred might be a more reasonable number for a model like EfficientNet.

That would be the difference between 4 days on a GPU like the GTX 1070, and only a few hours (6.5 to be precise) on a dual RTX 6000 Ada setup. Then consider that a real dataset doesn’t have only 2188 images, it could have millions (for reference ImageNet has just over 14 million images).

Industry progress

Another thing to bear in mind is progress in industry. EfficientNet is a few years old now, and things have moved on.

As a small example take NoisyStudent, which builds on the standard EfficientNets with a variation called EfficientNet-L2 and states:

Due to the large model size, the training time of EfficientNet-L2 is approximately five times the training time of EfficientNet-B7

-Self-training with Noisy Student improves ImageNet classification

…so speed really does matter if you need to stay at the cutting edge.

What does that mean for pro vs consumer graphics cards then?

The truth is that if you only look at speed of execution there is very little difference between professional and consumer GPUs if you compare like for like. An RTX 4090 is near as makes no difference the same speed as an RTX 6000 Ada.

An RTX 4090 is near as makes no difference the same speed as an RTX 6000 Ada.

All this little experiment has illustrated so far is that speed is very important, as industry standard models are progressing in complexity quite quickly. Older generation graphics cards are noticeably slower already. To keep up requires at least staying on the cutting edge of hardware.

…scale matters a great deal when answering this question.

…but with the speed of progression (just look at the rapid accent of GTP-3 and GTP-4) it also appears that if you want to stay at the cutting edge, one GPU, even at the level of the RTX 4090 or RTX 6000 Ada, is unlikely to be enough. If that is the case, then the superior cooling, less power draw and more compact size of the professional level graphics cards are a significant advantage when building a system.

Essentially, scale matters a great deal when answering this question.

However, speed is only one facet. Now let’s move on to the GPU RAM, where things get a little more interesting…

Discussion — GPU RAM

GPU RAM is a significant consideration in some situations, and can be a literal limiting factor as to whether certain models, or datasets, can be utilised at all.

Let’s see the pair of RTX 6000 Ada in full flow:

The two RTX 6000 Ada GPUs running a deep learning model. Image by Author

You may notice in the image above that the GPU RAM is at 100% for both GPUs. However, the this is not the real usage:

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to *CUDA_VISIBLE_DEVICES*) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.

-tensorflow.org

The limits

The absolute limit is brought home quite starkly by the fact that the GTX 1070 (which has 8GB of GPU RAM) is only capable of running EfficientNet B7 with a batch size of 1 (i.e. it can process 1 image at a time before having to update the model parameters and load the next image into the GPU RAM).

This causes two problems:

You lose speed of execution due to frequent parameter updates in addition to loading in fresh data to the GPU RAM more regularly (i.e. larger batch sizes are inherently quicker.)
If the input image size gets any larger, the model will not be able to run at all, as it won’t fit one single image into the GPU RAM

Even the Tesla T4 which has a not too shabby 16GB of GPU memory only manages a batch size of 2 on EfficientNet B7.

As detailed earlier, 16GB of GPU RAM is a good representation of the majority of current generation consumer GPUs, with only the RTX 4090 having more at 24GB. So this is a fairly significant downfall for consumer GPUs if you are dealing with memory heavy raw data.

At this point it suddenly becomes clear why all the professional GPUs are so RAM heavy when compared to their consumer equivalents. As mentioned in the discussion for the speed of execution, EfficientNet is no longer at the bleeding edge, so the reality today is probably even more demanding than outlined in the tests for this article.

System density

Another consideration in regard to GPU RAM is system density.

For example, the system I have been given access to has a motherboard that can take 4 double height GPUs (I have also seen systems with up to 8 GPUs). This means that if GPU RAM is a priority in your system, then professional GPUs are a no brainer:

4 x RTX 6000 Ada = 192GB GPU RAM and 1200W of power draw

4 x RTX 4090 = 96GB GPU RAM and 1800W of power draw

(…and as I have already mentioned earlier in the article the RTX 4090 is a triple slot GPU so this isn’t even realistic. In reality only two RTX 4090 graphics cards would actually fit, but for the sake of easy comparison let’s assume it would work.)

That is no small difference. To match the RTX 6000 Ada system in terms of GPU RAM you would need two separate systems drawing at least three times the power.

To match the RTX 6000 Ada system in terms of RAM you would need two separate systems drawing at least three times the power.

Don’t forget that as you would need two separate systems, you would have to fork out for additional CPUs, power supplies, motherboards, cooling, cases etc.

A side note on system RAM…

Did you notice 8 sticks of 64GB system RAM above and below the CPU in the professional system? Image via Exxact Corporation under license to Michael Clayton

It is also worth pointing out, that it is not just the GPU RAM that matters. As the GPU RAM scales up you need to increase the system RAM in parallel.

You may note in the Jupyter notebooks for the Tesla T4 that I have commented out the following optimisations:

train_data = train_data.cache().prefetch(buffer_size=tf.data.AUTOTUNE)
val_data = val_data.cache().prefetch(buffer_size=tf.data.AUTOTUNE)

This is because, for EfficientNet B7, the training will crash if they are enabled.

Why?

Because the “.cache()” optimisation keeps the data in system memory to feed it efficiently to the GPU, and the Colab instance only has 12GB of system memory. Which is not enough, even though the GPU RAM peaks at 9.9GB:

This [.cache()] will save some operations (like file opening and data reading) from being executed during each epoch.

-tensorflow.org

However, the professional system has 8 sticks of 64GB system RAM, for a total of 512GB of system RAM. So even though the two RTX 6000 Ada GPUs combined have 96GB of GPU RAM, there is still plenty of overhead in the system RAM to deal with heavy caching.

Conclusion

Image via Exxact Corporation under license to Michael Clayton

So, are professional level graphics cards better than consumer cards for deep learning?

Money no object. Yes, they are.

Does that mean that you should discard considering consumer level graphics cards for deep learning?

No, it doesn’t.

It all comes down to specific requirements, and more often than not scale.

Large datasets

If you know that your workload is going to be RAM intensive (large language models, image, or video based analysis for example) then professional graphics cards of the same generation and processing speed tend to have roughly double the GPU RAM.

It all comes down to specific requirements, and scale.

This is a significant advantage, especially considering there is no elevation in energy requirements to achieve this compared to a consumer graphics card.

Smaller datasets

If you don’t have high RAM requirements, then the question is more nuanced and relies on whether reliability, compatibility, support, energy consumption, and that additional 10% in terms of speed are worth the quite significant hike in price.

Scale

If you are about to invest in significant infrastructure, then reliability, energy consumption and system density may move from low priority to quite significant considerations. Areas that professional GPUs excel at.

Conversely, if you need a smaller system, and high GPU RAM requirements aren’t important, then considering consumer level graphics cards may turn out to be beneficial. Factors associated with large scale, such as reliability and energy consumption will become less of an issue, and system density won’t matter at all.

The final word

All in all it is a balancing act, but if I had to pick two items to summarise the most important factors in choosing between a consumer GPU and professional GPU it would be:

GPU RAM
System scale

If you have either high GPU RAM requirements, or will need larger systems with multiple GPUs, then you need a professional level GPU/GPUs.

Otherwise, most likely, consumer level will be a better deal.

Encrypted Arch Linux Installation Guide

Thu, 30 May 2024 01:00:00 GMT

Arch Linux is probably avoided by many due to the initial hurdle of actually getting it installed, which is understandable. This is unfortunate, because once it is installed it is probably the most sorted operating system I have ever used, and not unstable at all in my experience.

It has excellent documentation with the Arch Wiki, something I had come across time and again, even before using Arch. It also has a knowledgable and active community, reflected by the enormity of the AUR. And to round things of, you are always bang up to date with the latest kernel and packages.

Hopefully, this guide will help you overcome that initial hurdle so you can see for yourself.

Best of luck!

This article was last updated on 7th January 2026

Introduction

This guide provides information on how to setup an Arch Linux installation from scratch with the following features:

Encrypted root, swapfile and boot (only the EFI partition is unencrypted)
Btrfs file system and subvolumes setup
Encrytped swapfile (rather than swap partition), including the necessary settings for hibernation
Gnome desktop installation
NVIDIA proprietary driver setup
Wayland/Gnome(gdm) configuration for NVIDIA
Snapper snapshot installation and configuration, including btrfs-assistant GUI

To put this guide together I have referenced various blog posts, and of course the Arch Wiki. Specific sources are detailed in the references section at the end.

This guide should produce a complete and functional system, with GUI and snapshot capability.

This article is also available as a GitHub repo. Therefore, if you have any suggestions for improvements to any of the methods used in this guide, please feel free to put in a bug report (or pull request) on GitHub.

I am aware that preferred / recommended methods change regularly, so I have tried to stick closely to the Arch Wiki where possible.

Note: this guide at some point may become outdated, the arch wiki should always be considered the main and trusted source for any information.

Pre-requisites

It is assumed that you have already downloaded Arch Linux, burned it to a USB drive, and booted into the live environment.

If you are now sitting looking at the Arch live environment commandline then you are ready to go, otherwise please get setup first.

You may also need buckets of patience...best of luck!

Set the console keyboard layout and font

The default console keymap is "US". Available layouts can be listed with:

localectl list-keymaps

To set the keyboard layout, pass the keyboard layout name to loadkeys. For example, to set a UK keyboard layout:

loadkeys uk

Console fonts are located in /usr/share/kbd/consolefonts/ and can be set with setfont, omitting the path and file extension. For example, to use one of the largest fonts suitable for HiDPI screens, run:

setfont ter-132b

Check if the system is UEFI

If the following command returns 64, then the system is booted in UEFI mode, and has a 64-bit x64 UEFI. If the command returns 32, then system is booted in UEFI mode and has a 32-bit IA32 UEFI. Either of which is fine to proceed with this guide, as we will be using GRUB.

If the file doesn't exist then the system isn't UEFI, so you won't be able to proceed.

cat /sys/firmware/efi/fw_platform_size

Connect to Wifi

If your device is plugged in via Ethernet cable then you should be good to go. Otherwise, connect to a Wi-Fi network using iwctl:

# Enter the iwctl interface
iwctl
# Find the name of your wireless device:
device list
# Scan for networks:
station <device name> scan
# List network SSID:
station <device name> get-networks
# Connect to network:
station <device-name> connect <SSID>

Leave iwctl by pressing Ctrl+C.

Test the connection:

ping archlinux.org

If the ping responds, then stop the process using Ctrl+C.

Update the system clock

Enable and start network time synchronisation:

# check current settings
timedatectl
# list timezones (change "Europe/" as appropriate)
timedatectl list-timezones | grep "Europe/"
# or if you want to list everything omit the 'grep'
timedatectl list-timezones
# set timezone (change "Europe/London" to your timezone)
timedatectl set-timezone "Europe/London"
# turn on ntp
timedatectl set-ntp true
# check the settings have updated
timedatectl

Prepare the drive

Disks are assigned to a block device such as /dev/sda, dev/nvme0n1 or /dev/mmcblk0. Let's list out the devices:

fdisk -l

Wipe the disk

It is a good idea to wipe the disk before proceeding any further.

Create a container called "wipe_me".

Note: the "block-device" should be the main device not one of the device partitions. For example, it could be /dev/sdf , but not /dev/sdf1 , or /dev/nvme0n1 but not /dev/nvmen0n1p1.

NOTE: YOU ARE ABOUT TO WIPE THE DISK! BE SURE YOU DON'T NEED THE DATA ON THE DISK AS IT IS NOT RECOVERABLE!

cryptsetup open --type plain -d /dev/urandom /dev/<block-device> wipe_me

Zero out the container. This may take a while depending on the size and type of drive:

dd bs=1M if=/dev/zero of=/dev/mapper/wipe_me status=progress

Then close the container:

cryptsetup close wipe_me

Partition the disk

With the drive erased, use fdisk to partition the disk.

Using fdisk is an interactive process. Start the interactive process by telling fdisk the drive to setup.

Note: until you give the write command 'w' (write table to disk and exit), nothing will be changed on the disk. So if you make a mistake, just type 'q' (quit without saving changes), and start again.

# As per the pervious section the "block-device" should be the main device not one of the device partitions.
fdisk /dev/<block-device>

This is now the interactive part of the process with fdisk.

Enter 'm' to see the available commands:

Output:

  GPT
   M   enter protective/hybrid MBR

  Generic
   d   delete a partition
   F   list free unpartitioned space
   l   list known partition types
   n   add a new partition
   p   print the partition table
   t   change a partition type
   v   verify the partition table
   i   print information about a partition

  Misc
   m   print this menu
   x   extra functionality (experts only)

  Script
   I   load disk layout from sfdisk script file
   O   dump disk layout to sfdisk script file

  Save & Exit
   w   write table to disk and exit
   q   quit without saving changes

  Create a new label
   g   create a new empty GPT partition table
   G   create a new empty SGI (IRIX) partition table
   o   create a new empty MBR (DOS) partition table
   s   create a new empty Sun partition table

Start the partitioning

Two partitions will be created:

The EFI partition - this will be ONLY the EFI, and will not contain boot.
The root partition, which will contain everything else, including boot and swap (in our case a swapfile), and will be encrypted.

# CREATE A NEW PARTITION TABLE
g # create a new empty GPT partition table

# CREATE THE EFI PARTITION
n # add a new partition
*enter* # Partition number (1-128, default 1)
*enter* # First sector (2048-250069646, default 2048)
+512M # Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-250069646, default 250068991)

# Created a new partition 1 of type 'Linux filesystem' and of size 512 MiB.

t # change the partition type, as we need an EFI partition, not 'Linux filesystem'
uefi # 'uefi' is an alias for '1', so you can use either '1' or 'uefi' here

# Changed type of partition 'Linux filesystem' to 'EFI System'.

# CREATE THE ROOT PARTITION
n # add a new partition
*enter* # Partition number (2-128, default 2)
*enter* # First sector (1050624-250069646, default 3147776)
*enter* # Last sector, +/-sectors or +/-size{K,M,G,T,P} (1050624-250069646, default 250068991)

# Created a new partition 2 of type 'Linux filesystem' and of size 118.7 GiB.

# PRINT THE PARTITION TABLE
p # print the partition table

# Disk /dev/sdf: 119.24 GiB, 128035676160 bytes, 250069680 sectors
# Disk model: SSD 850 PRO 128G
# Units: sectors of 1 * 512 = 512 bytes
# Sector size (logical/physical): 512 bytes / 512 bytes
# I/O size (minimum/optimal): 512 bytes / 33553920 bytes
# Disklabel type: gpt
# Disk identifier: 63A0B976-C21F-47B8-B9FB-DD16BA279098

# Device       Start       End   Sectors   Size Type
# /dev/sdf1     2048   1050623   1048576   512M EFI System
# /dev/sdf2  1050624 250068991 249018368 118.7G Linux filesystem

# WRITE THE TABLE TO DISK
w # write table to disk and exit

Note: From now on I will use the partition names above as examples ( i.e. /dev/sdf1 and /dev/sdf2) , but they need to be changed as appropriate for your actual setup. For example with an nvme device the partition names will likely be in the format /dev/nvme0n1p1 and /dev/nvme0n1p2. Use whatever is in the "Device" column when you printed the partition table above.

Create the filesystems

There are two partitions, and each will have it's own filesystem. The root partition will use btrfs, but the EFI partition cannot use btrfs, and so will be set as FAT.

Create the EFI filesystems

mkfs.vfat -F 32 /dev/sdf1

Encrypt and open the root partition

You must use luks1 here - Currently, the latest grub does support opening a luks2 partition, but it does not support the argon2id encryption algorithm yet. So to have an encrypted boot requires luks1 for the moment.

Note: I have upped the iteration parameter to 5000 from the default 3000. This will result in the unlocking of encrypted partition taking a little longer at boot (nothing excessive, but it depends on hardware). If this is a problem (i.e. you have a slow processor), please change the 5000 in the command below back to 3000. I would not recommend going any lower than the default of 3000.

# You will be asked to create a password. Make sure it is a strong password
cryptsetup --type luks1 -c aes-xts-plain64 -h sha512 -i 5000 -s 512 luksFormat /dev/sdf2
# Now open the new encrypted partition. You will be prompted to enter the password you created in the previous step.
cryptsetup luksOpen /dev/sdf2 cryptroot

Create the root filesystem

mkfs.btrfs -L archlinux /dev/mapper/cryptroot

Mount the root device

mount /dev/mapper/cryptroot /mnt

Configuring btrfs

As btrfs is being used, then subvolumes should be created.

Create btrfs subvolumes

Various subvolumes will be created, this is mainly to help with snapshots later in the process.

When taking a snapshot of root, the other subvolumes will not be included in the snapshot. (This is useful, for example, if you want to access logs after restoring a previous snapshot, as @log stops the logs from being rolled back along with root).

If you don't want so many, then you must at least keep the first four below (@cache, @log and @tmp are not strictly necessary).

btrfs subvolume create /mnt/@
btrfs subvolume create /mnt/@swap
btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@snapshots
btrfs subvolume create /mnt/@cache
btrfs subvolume create /mnt/@log
btrfs subvolume create /mnt/@tmp

Unmount the root partition

umount /mnt

Set the options for subvolume mounting

# set subvolume options for main btrfs subvolumes
export opts="ssd,noatime,compress=zstd:1,space_cache=v2,discard=async"

Options:

ssd - specification of the allocation scheme suitable for SSD drives (rather than rotational drives).
noatime - significantly improves read intensive workload performance, and also reduces writes in some circumstances.
compress=zstd:1 - zstd level 1 compression is optimal for NVME devices, while gaining some compression. If compression is increased too much with rapid NVME drives, the compression can become a bottleneck. Omit the :1 to use the default compression level of 3. zstd accepts a value range of 1-15, with higher levels trading speed and memory for higher compression ratios.
space_cache=v2 - creates cache in memory for greatly improved performance. Be sure to use v2 and not v1.
discard=async - keeps the disk tidy by enabling the discarding of freed file blocks. Specifically, the asynchronous mode (async) gathers extents in larger chunks before sending them to the devices for TRIM.

Please take a look at the official documentation for further details.

Mount the root BTRFS subvolume

mount -o ${opts},subvol=@ /dev/mapper/cryptroot /mnt

Create mountpoints for other BTRFS subvolumes

Currently the root partition doesn't contain any subfolders to mount the other partitions to, so they need to be created.

mkdir -p /mnt/{swap,home,.snapshots,var/cache,var/log,var/tmp}

Mount the other subvolumes (except swap)

Mount the previously created btrfs subvolumes to the folders created in the previous section.

mount -o ${opts},subvol=@home /dev/mapper/cryptroot /mnt/home
mount -o ${opts},subvol=@snapshots /dev/mapper/cryptroot /mnt/.snapshots
mount -o ${opts},subvol=@cache /dev/mapper/cryptroot /mnt/var/cache
mount -o ${opts},subvol=@log /dev/mapper/cryptroot /mnt/var/log
mount -o ${opts},subvol=@tmp /dev/mapper/cryptroot /mnt/var/tmp

Mount swap

The swap subvolume needs different mounting options as it cannot use COW (Copy-On-Write), and hence cannot use compression:

mount -o noatime,ssd,subvol=@swap /dev/mapper/cryptroot /mnt/swap

Setup swap

The two lines below are sufficient to create a swapfile with all the necessary correct settings (such as nocow). See the official guidance for more details.

Note: If you want to use hybernation you must assign a swap file that is at least as large as your RAM.

# create swapfile - change the 62g to your required swap filesize (62g = 62 Gigabytes)
btrfs filesystem mkswapfile --size 62g --uuid clear /mnt/swap/swapfile
# activate swapfile
swapon /mnt/swap/swapfile

Mount EFI and boot

The EFI is mounted on the first partition, which is formated with FAT. This guide uses /efi rather than the quite common /boot/efi. This follows the recommendation of the Arch wiki, and also makes sense since in this instance the UEFI info is on a completely separate partition to boot.

The boot directory just needs creating, not mounting, as the root directory is already mounted, which is where boot will reside.

mkdir /mnt/efi
mkdir /mnt/boot
mount /dev/sdf1 /mnt/efi

Syncronise the Package Database

pacman -Syy

Update the Mirror List

Below, swap out the countries for those of your choice.

reflector --verbose --protocol https --latest 5 --sort rate --country 'Sweden,Monaco,Switzerland,Germany' --save /etc/pacman.d/mirrorlist

Install the Arch Base System

Add to, or change, the listed programs as you wish. However, this is a good starting point.

Be sure that if you remove anything you know what you are doing, as some packages are required later on in the install process.

For example, you may wish to switch out intel-ucode for amd-ucode if you have an AMD CPU. Also you could add an alternative text editor such as nano. However, even if you add an alternative text editor please don't remove vim as it is required later for using visudo.

You can install other packages later, so you don't need to go crazy here.

Note: an lts kernel will be installed as an option later, so there is no need to do it here.

pacstrap /mnt base base-devel linux linux-headers linux-firmware intel-ucode btrfs-progs grub efibootmgr vim networkmanager gvfs exfatprogs dosfstools e2fsprogs man-db man-pages texinfo openssh git reflector wget cryptsetup wpa_supplicant terminus-font

Generate fstab

This will autogenerate the fstab based on the subvolumes that have been created so far.

genfstab -U /mnt >> /mnt/etc/fstab

Check result with:

cat /mnt/etc/fstab

Remove subvolid

As per the Arch wiki for btrfs:

One can mimic traditional file system partitions by creating various subvolumes under the top level of the file system and then mounting them at the appropriate mount points. It is preferable to mount using subvol=/path/to/subvolume, rather than the subvolid, as the subvolid may change when restoring #Snapshots, requiring a change of mount configuration.

-Arch wiki

Therefore if you want to use something like timeshift or snapper later, then you should make this adjustment.

# completely remove the "subvolid" entries from fstab
vim  /mnt/etc/fstab

Enter the new system

We will now 'chroot' into the newely installed system.

arch-chroot /mnt /bin/bash

Basic Settings

Now we are operating in the new install, so let's start by setting some basics.

Set system clock

Change the 'Europe/London' as appropriate for your timezone.

ln -s /usr/share/zoneinfo/Europe/London /etc/localtime
hwclock --systohc

Set hostname

Change 'your_hostname' to your actual hostname. (This can be anything you want, it is essentially what you want your computer to be named.)

echo your_hostname > /etc/hostname

Set hosts

Again, change 'your_hostname' to be the same as the hostname you set in the previous step.

cat > /etc/hosts <<EOF
127.0.0.1   localhost
::1         localhost
127.0.1.1   your_hostname.localdomain your_hostname
EOF

Set Locale

Remember to change the relevant parts to your specific locale.

Option 1

export locale="en_GB.UTF-8"
sed -i "s/^#\(${locale}\)/\1/" /etc/locale.gen
echo "LANG=${locale}" > /etc/locale.conf

Option 2

Alternatively, go in and edit the files directly.

In /etc/locale.gen uncomment en_GB.UTF-8 UTF-8:

en_GB.UTF-8 UTF-8

Add the following line to /etc/locale.conf:

LANG=en_GB.UTF-8

Generate the locales

The locales you just created need to be generated. To do this run the below:

locale-gen

Set keyboard layout and terminal font

Change "uk" to whatever your keyboard layout is.

echo "KEYMAP=uk" >> /etc/vconsole.conf
echo "FONT=ter-v28n" >> /etc/vconsole.conf

Other font sizes if 28 is too small or large for you:

ter-v20n
ter-v24n
ter-v32n

(also change the 'n' to 'b' if you want 'bold' rather than 'normal')

Set default editor

Swap in your favourite editor here (e.g. nano rather than vim)

echo "EDITOR=vim" >> /etc/environment
echo "VISUAL=vim" >> /etc/environment

Set accounts and passwords

Some accounts and associated passwords should now be created

Set the root password

The root account will already exist, but it doesn't have a password yet.

passwd

Create a user and user password

This will be your user account, so swap in your username where it says user_name and set your account password.

The below will also add your user to the wheel group, which will allow the user to use sudo commands in conjunction with their password.

# add user
useradd -m -G wheel -s /bin/bash user_name
# set password
passwd user_name

Activate wheel

It is recommended to never adust /etc/sudoers directly, as any mistakes can permanently bork your system. visudo is a command designed specifically to edit this file, and has checks in place to ensure that any mistakes are caught.

# issuing this command will open /etc/sudoers with vim
visudo

Now uncomment this line:

## Uncomment to allow members of group wheel to execute any command
# %wheel ALL=(ALL:ALL) ALL

so it becomes:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL:ALL) ALL

Create the crypto keyfile

This section will create an encrypted keyfile. The reason this is necessary is to remove the requirement to supply the encryption unlock password twice at boot.

With the keyfile, the password is supplied once, and the system will be completely decrypted.

cd /
dd bs=512 count=4 if=/dev/random of=crypto_keyfile.bin iflag=fullblock
chmod 000 /crypto_keyfile.bin
chmod 600 /boot/initramfs-linux*
cryptsetup luksAddKey /dev/sdf2 /crypto_keyfile.bin
# enter your encryption password when prompted
cryptsetup luksDump /dev/sdf2
# You should now see that LUKS Key Slots 0 and 1 are both occupied

Configure mkinitcpio

mkinitcpio is a script that creates the initial ramdisk. It basically sets up various kernel modules, and performs initialisation steps.

Open the conf file:

vim /etc/mkinitcpio.conf

Edit the following:

# load the keyfile we created
FILES=(/crypto_keyfile.bin)

# Remove "kms" from hooks if you use a NVIDIA card so that you can use proprietary drivers later
# Add "resume" if you want to use hibernation later
# Add "encrypt" before "filesystem"
# the below is my suggested ordering, but it could be adjusted if you know what you are doing
HOOKS=(base udev keyboard autodetect keymap modconf microcode consolefont block encrypt resume filesystems fsck)

# Add "crc32c". Note: there used to be an option for "crc32c-intel" if you have an intel CPU which supports SSE4.2, but this is now deprecated.
MODULES=(btrfs crc32c)

Recompile initcpio:

mkinitcpio -p linux

Configure Grub

Now we will go through a few steps to setup grub so the system can boot.

Get UUID of encrypted partition

blkid -s UUID -o value /dev/sdf2
# output:
# 180901b5-151a-45e3-ba87-28f02b124666

Get UUID of root

blkid -s UUID -o value /dev/mapper/cryptroot
# output:
# aba2d556-2891-4f79-a78e-4f9a6e439b02

Get swapfile offset for resume

This must be done as per the below for btrfs (using the 'filefrag' method will be inaccurate):

btrfs inspect-internal map-swapfile -r /swap/swapfile
# output:
# 533760

Update the grub file settings

Open the grub setting file:

vim /etc/default/grub

Add the following to what already exists. Note that the UUID for the encrypted partition is used for the "cryptdevice" and the UUID for root is used for "resume".

If you don't need hibernation you can remove "resume" and "resume_offset"

GRUB_CMDLINE_LINUX_DEFAULT="cryptdevice=UUID=180901b5-151a-45e3-ba87-28f02b124666:cryptroot resume=UUID=aba2d556-2891-4f79-a78e-4f9a6e439b02 resume_offset=533760"
GRUB_PRELOAD_MODULES="luks"
# This exists, but needs turning on (i.e. remove the initial #)
GRUB_ENABLE_CRYPTODISK=y

Install grub

There are two options for running grub install. My recommendation is to use option 2 below, but it depends on your preference.

If you want some more information on why you might want to use one over the other then there is some useful information here. Specifically take a look at the last paragraph of the section "The EFI Boot Process".

Option 1: Standard install method

This method is the standard way to install grub. This will produce an entry in your UEFI (efi) directory, and will be loaded into NVRAM by your BIOS when the system boots.

I have had issues with this method in the past, as in some circumstances (for example swapping out hard drives, or moving a drive to a new system), it can cause the boot entries in the BIOS to disappear, making the drive unbootable until this setup is re-run.

There are also instances of some (probably older) motherboards not recognising this method, so the drive doesn't appear in BIOS.

grub-install --target=x86_64-efi --efi-directory=/efi --bootloader-id=Arch

Note: the --bootloader-id can be whatever you want. This is the name that will be shown in bios/grub so you can identify the install.

Verify that a GRUB entry has been added to the UEFI bootloader by running:

efibootmgr

(It will be the one with the name specified by "--bootloader-id" from grub-install).

Option 2: Removable install method

This is my preferred method, as it avoids the issues of the first method by installing the bootloader info in a specific standard location: EFI/BOOT/bootx64.efi. This means you are not relying on NVRAM, and so the entry will never get "lost" or removed by the BIOS.

Typically this is the method used by bootable USB drives, but is perfectly fine for normal drives that you have no intention of removing.

grub-install --target=x86_64-efi --efi-directory=/efi --removable

Generate the GRUB configuration file

grub-mkconfig -o /boot/grub/grub.cfg

Verify that grub.cfg has entries for insmod cryptodisk and insmod luks by running:

grep 'cryptodisk\|luks' /boot/grub/grub.cfg

Exit and reboot

exit
swapoff -a
umount -R /mnt
reboot

After reboot you should end up at the arch commandline login screen (fingers crossed!).

After reboot checks

# Failed systemd services
sudo systemctl --failed

# High priority errors in the systemd journal
# There may be items here, it doesn't mean the install failed
# For example I had broadcom bluetooth fails, which is normal for my system
sudo journalctl -p 3 -xb

# Check swap is active (various methods):
free -m
swapon --show
cat /proc/swaps

Connect to wifi (if required)

# Start NetworkManager
sudo systemctl enable NetworkManager.service
sudo systemctl start NetworkManager.service

# Connect to WIFI
nmcli device wifi list
nmcli device wifi connect SSID_or_BSSID password your_password

# Check WIFI Connection
nmcli connection show

Pacman config (optional)

sudo vim /etc/pacman.conf

Uncomment (or add) the following if you wish. "Color" will provide color in the terminal, and "ILoveCandy" will make the loading bar into a "pacman" like animation.

# Misc options
Color
ILoveCandy

Uncomment the following to allow parallel downloads:

ParallelDownloads = 5

You can change the number 5 to whatever you like depending on your connection speed.

Update system

sudo pacman -Syyu

Install LTS kernal (recommended, but optional)

Install the Long-Term Support (LTS) Linux kernel as a fallback option to Arch's default kernel.

As arch is generally at the bleeding edge, it can mean that there is more chance of conflicts arising with a specific new kernel. One way to mitigate this is to have an lts kernel to fall back to. Hence, the recommendation.

Note: this will not replace the 'normal' kernel, you will have a choice between 'normal' and 'lts' at boot.

sudo pacman -S linux-lts linux-lts-headers
sudo grub-mkconfig -o /boot/grub/grub.cfg

Reboot and select LTS kernel to test.

After reboot using the LTS kernel, confirm that the running kernel is indeed lts

uname -r

You may find that the LTS kernel is now the first in the boot order. This will result in the LTS kernel being automatically loaded every time you boot, rather than the normal kernel, unless you intervene at the boot menu.

This is inconvenient and can get quite annoying.

Until quite recently there was no good option to remedy this situation, although there were a few slightly 'hacky' ways to get around it. However, with a recent update of grub there is now a much better solution as described in the Arch wiki.

First open up the following file:

sudo vim /etc/default/grub

Once you have the file grub open as per the above, the following line should be added as the top entry in the file:

GRUB_TOP_LEVEL="/boot/vmlinuz-linux"

This will change the order of the kernels in the selection menu on boot, essentially putting the normal kernel above the LTS kernel.

Then regenerate grub:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Install pipewire for sound

sudo pacman -S pipewire pipewire-alsa pipewire-pulse pipewire-jack wireplumber alsa-utils

Then reboot.

Install yay for access to the AUR

git clone https://aur.archlinux.org/yay-git.git
cd yay-git
makepkg -si

Install GNOME

I'm aware that a desktop environment is a personal thing, so you can of course install whatever you want here if you don't like GNOME. However, just bear in mind that I have not tested this guide with any other desktop environment, so if you hit problems you will need to get creative!

sudo pacman -S gnome gnome-extra
sudo systemctl enable gdm.service

Then reboot.

NOTE: on initial reboot I was met with a black screen immediately after providing my user password at the Gnome login screen, and was unable to proceed to the Gnome desktop. A simple reboot alleviated this. Whether you will experience the same I am not sure.

After reboot you should have a fully functional Gnome desktop environment using wayland. However, there are various other items that can be setup as well, which the next few sections will go through:

Setup a NVIDIA graphics card with proprietary drivers from NVIDIA
Disable root account for additional security
Setup snapper to allow snapshots to be taken (including GUI)

NVIDIA Setup (Optional)

If you have a NVIDIA graphics card then this section is recommended.

Enable the multilib repository

Open pacman conf:

sudo vim /etc/pacman.conf

Uncomment the following lines by removing the # character at the start of them

[multilib]
Include = /etc/pacman.d/mirrorlist

Update the package list, and install packages with yay.

NOTE: As of January 2026 NVIDIA has deprecated all cards from the 10xx series and lower (e.g. 1070, 1080 etc.). If you own one of these cards you will need to use packages from the AUR. See this post on archlinux.org for more details.

NVIDIA 20xx and 1650 series and higher:

yay -Syu
yay -S nvidia nvidia-lts nvidia-utils libxnvctrl opencl-nvidia lib32-nvidia-utils nvidia-settings

NVIDIA 10xx series and lower:

yay -Syu
yay -S nvidia-580xx-dkms nvidia-580xx-utils nvidia-580xx-settings opencl-nvidia-580xx lib32-nvidia-580xx-utils libxnvctrl-580xx

Set enviromental variables

Force the GBM (Generic Buffer Management) backend.

sudo vim /etc/environment
# Append
GBM_BACKEND=nvidia-drm
__GLX_VENDOR_LIBRARY_NAME=nvidia

DRM Settings

Enable DRM (Direct Rendering Manager)

From the Arch Wiki:

Since NVIDIA does not support automatic KMS late loading, enabling DRM (Direct Rendering Manager) kernel mode setting is required to make Wayland compositors function properly.

-Arch Wiki

Option 1 - Add parameters using modprobe (recommended method)

This is the preferred method, as per the Arch Wiki:

To enable DRM (Direct Rendering Manager), set the modeset=1 kernel module parameter for the nvidia_drm module...

...Additionally, with driver version 545 and above, you can also set the experimental nvidia_drm.fbdev=1 parameter, which is required to tell the NVIDIA driver to provide its own framebuffer device instead of relying on efifb or vesafb, which do not work under simpledrm...

...nvidia_drm.fbdev=1 has known issues that are only possibly fixed in the driver version 550 and above.

echo -e 'options nvidia_drm modeset=1 fbdev=1' | sudo tee -a /etc/modprobe.d/nvidia.conf

Other parameters

There are also other (non-essential) parameters that can be set at the same time. There is a great rundown of potential parameter settings on the Gentoo Wiki, but I have selected a relevant few to apply here (the bullet point explanation text below is quoted from the Gentoo Wiki):

NVreg_UsePageAttributeTable --> Default:0 - This is one of the latest and newest additions to the NVIDIA driver modules option. It allows the driver to take full advantage of the PAT technology - a newer way of allocating memory, replacing the older Memory Type Range Register (MTRR) method. The PAT method creates a partition type table at a specific address mapped inside the register and utilizes the memory architecture and instruction set more efficiently and faster. If the computer supports PAT and the feature is enabled in the kernel then this flag can be enabled. Without PAT support, users may experience unstable performance and even crashes if this is enabled. So be careful with these options.
NVreg_InitializeSystemMemoryAllocations --> Default:1 - Tell the NVIDIA driver to clear system memory allocations prior to using it for the GPUs. Disabling can give a slight performance boost but at the cost of increased security risks. By default the driver will wipe the allocated by zeroing out its content.
NVreg_PreserveVideoMemoryAllocations --> Default:0 - By default the NVIDIA Linux drivers save and restore only essential video memory allocations on system suspend and resume. Quoting NVIDIA "The resulting loss of video memory contents is partially compensated for by the user-space NVIDIA drivers, and by some applications, but can lead to failures such as rendering corruption and application crashes upon exit from power management cycles.". The "still experimental" interface enables saving all video memory (given enough space on disk or RAM).

If you want to apply these parameters, do the following:

echo -e 'options nvidia NVreg_UsePageAttributeTable=1 NVreg_InitializeSystemMemoryAllocations=0 NVreg_PreserveVideoMemoryAllocations=1' | sudo tee -a /etc/modprobe.d/nvidia.conf

Option 2 - update grub (alternate method - NOT recommended)

As an alternative to the previous modprobe method parameters can also be passed to the Linux kernel during its initial boot, through the GRUB bootloader.

Open the grub settngs file:

sudo vim /etc/default/grub

Add the nvidia-drm.modeset=1' to GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="nvidia_drm.fbdev=1 nvidia_drm.modeset=1 loglevel=3 quiet..."

Regenerate the grub config:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Update mkinitcpio

sudo vim /etc/mkinitcpio.conf

Append the following to what is already there:

MODULES=(nvidia nvidia_modeset nvidia_uvm nvidia_drm)

# Add this if you used the recommended method (option 1) from the previous section
FILES=(/etc/modprobe.d/nvidia.conf)

# If you haven't already, please also remove "kms" from "HOOKS=()"

Removing kms from HOOKS=() ensures that the initramfs will avoid including the open-source “nouveau” driver, which may conflict with the proprietary NVIDIA drivers.

Recompile:

sudo mkinitcpio -P

Preserve video memory after suspend

By default the NVIDIA Linux drivers save and restore only essential video memory allocations on system suspend and resume. Quoting NVIDIA:

The resulting loss of video memory contents is partially compensated for by the user-space NVIDIA drivers, and by some applications, but can lead to failures such as rendering corruption and application crashes upon exit from power management cycles.

-nvidia.com

The "still experimental" interface enables saving all video memory (given enough space on disk or RAM).

sudo systemctl enable nvidia-suspend.service nvidia-hibernate.service nvidia-resume.service

Add Pacman Hook

This will automatically update initramfs after a NVIDIA upgrade.
Arch wiki source

sudo mkdir /etc/pacman.d/hooks
sudo vim /etc/pacman.d/hooks/nvidia.hook

Add the following to the file:

[Trigger]
Operation=Install
Operation=Upgrade
Operation=Remove
Type=Package
Target=nvidia
Target=nvidia-lts
Target=linux
Target=linux-lts

[Action]
Description=Updating NVIDIA module in initcpio
Depends=mkinitcpio
When=PostTransaction
NeedsTargets
Exec=/bin/sh -c 'while read -r trg; do case $trg in linux*) exit 0; esac; done; /usr/bin/mkinitcpio -P'

Force enable wayland

The udev rules contained in 61-gdm.rules need to be turned off to force wayland to work with NVIDIA graphics.

You will likely find that 61-gdm.rules does not exist in /etc/udev/rules.d/, which is normal, and what you want.

The following command creates a "symbolic link" to "no information" (i.e. /dev/null). This effectively overrides /usr/lib/udev/rules.d/61-gdm.rules (which does exist), as /etc/udev takes precedence over /usr/lib/.

NOTE: it is not a good idea to adjust the file in /usr/lib as it will be overwritten automatically on updates.

sudo ln -s /dev/null /etc/udev/rules.d/61-gdm.rules

Reboot and check

# Regenerate initramfs:
sudo mkinitcpio -P
# Reboot
reboot

NOTE: as with the reboot after installing Gnome, on initial reboot I was met with a black screen immediately after providing my user password at the Gnome login screen, and was unable to proceed to the Gnome desktop. A simple reboot alleviated this. Whether you will experience the same I am not sure.

After reboot you can check if NVIDIA drm settings were applied with:

sudo cat /sys/module/nvidia_drm/parameters/modeset

It should return Y

Disable the root account (Optional)

As per the Arch Wiki:

Users may wish to disable the root login. Without root, attackers must first guess a user name configured as a sudoer as well as the user password.

-Arch Wiki

sudo passwd -l root

Note: be careful here, as you could lock yourself out! Make sure you added your user to the wheel group, and activated wheel using visudo (all of this was covered earlier in this install guide).

Enable snapshots with snapper

There are basically two options when it comes to making snapshots with a Btrfs filesystem:

Timeshift
Snapper

I have used timeshift for many years, and it has saved my skin quite a few times. It just works. However, it is not as flexible / configurable as snapper, so in this instance snapper will be installed.

Basic setup

This section comes across as a mess of deleting and recreating things, but unfortunately this is necessary because of the way the snapper 'create_config' currently works.

# install snapper
sudo pacman -S snapper snap-pac

# remove the original snapshots folder as the snapper script will want to create it's own
sudo umount /.snapshots
sudo rm -rf /.snapshots

# run config creation for the root filesystem
sudo snapper -c root create-config /

# list the btrfs subvolumes after config
sudo btrfs subvolume list /

# in the subvolumes list you will note that snapper has created ".snapshots", but also that our original "@snapshots" still exists. We don't need both.
# remove the subvolume created by snapper, so we can use "@snapshots"
sudo btrfs subvolume delete /.snapshots
# re-make the directory we deleted earlier and mount it
sudo mkdir /.snapshots
sudo mount -a
# update the permissions for the new folder
sudo chmod 750 /.snapshots

Manual Snapshot

Snapper is essentially setup, so an initial snapshot can be taken here if you wish.

# example syntax for creating a manual snapshot
sudo snapper -c root create -d "First snapshot"

Automatic Timeline Snapshots

Open the config for "root":

sudo vim /etc/snapper/configs/root

Set the following:

# required
ALLOW_USERS="your_username"
ALLOW_GROUPS="wheel"
# the below can be changed as you please
TIMELINE_MIN_AGE="3600"
TIMELINE_LIMIT_HOURLY="5"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="2"
TIMELINE_LIMIT_MONTHLY="1"
TIMELINE_LIMIT_QUARTERLY="0"
TIMELINE_LIMIT_YEARLY="0"

Enable services:

sudo systemctl enable --now snapper-timeline.timer
sudo systemctl enable --now snapper-cleanup.timer

How to list configs and snapshots

snapper list-configs
snapper -c root list

Skip indexing on .snapshots

It doesn't make sense to have indexing on the snapshots, so it may as well be disabled.

sudo pacman -S plocate
sudo vim /etc/updatedb.conf
# add the following
PRUNENAMES = ".snapshots"

Auto update Grub

This will allow the snapshots to appear, and be accessible from the GRUB menu on boot.

Install the grub-btrfs package:

sudo pacman -S grub-btrfs

If you have followed this guide then you don't need to change anything in the grub-btrfs config. However, if you have a different location for boot/grub then you may need to edit the following parameters in the config file:
GRUB_BTRFS_GRUB_DIRNAME="/boot/grub"
GRUB_BTRFS_BOOT_DIRNAME="/boot"

sudo vim /etc/default/grub-btrfs/config

Automatically update grub upon snapshot creation or deletion

sudo pacman -S inotify-tools
sudo systemctl enable grub-btrfsd
sudo systemctl start grub-btrfsd

Read-only snapshots and overlayfs

Typically, if you boot into a snapshot it will be readonly. This can cause problems (crashing/freezing) as some parts of the root directory need write access to funciton properly.

To get around this, 'overlayfs' can be used. Overlayfs allows any changes made to be temporarily saved in RAM.

This effectively gets around the write access issue, so no crashing/freezing. However, because it doesn't change the snapshot at all (the changes are in RAM), it also retains the integrity of the snapshot.

Note: all changes made are lost on reboot.

sudo vim /etc/mkinitcpio.conf
# add 'grub-btrfs-overlayfs' as the last item in HOOKS
HOOKS=(base ... fsck grub-btrfs-overlayfs)
# Regenerate initramfs
sudo mkinitcpio -P

Install btrfs-assistant (GUI)

btrfs-assistant provides an intuitive GUI interface for managing snapshots and settings for snapper. It makes restoring, creating and monitoring snapshots a breeze.

yay -S btrfs-assistant

The End

There are many more things that could be added to this guide, but I think here is a good place to stop.

You should now have a fully working Arch Linux install with:

Encrypted root, swapfile and boot
A btrfs filesystem with compression and snapshot capabilities
Gnome desktop using Wayland
NVIDIA proprietary drivers
Snapper snapshots including GUI
The license to say "I use Arch BTW!"

References

Below are links to some of the general references and blog post used to create this guide.

Arch install guidance

These two were absolutely essential for this guide:

There are also plenty of other guides that helped fill in the blanks here and there, but be careful as things can be outdated. If you are unsure, the Arch wiki should always be your main reference:

NVIDIA settings guidance

Btrfs

Some good guidance and comparisons are made here to help with a decision on what is appropriate for you in terms of Btrfs compression:

thelinuxcode - btrfs compression

Swapfile info:

Btrfs official docs - swapfile info

Snapper setup

How To Encrypt a Disk Using LUKS and Automount on Boot

Tue, 30 Dec 2025 00:00:00 GMT

Encryption should really be used wherever possible. There is no reason to have your data easily accessible to anybody that might be snooping around, whether you think the data is sensitive or not.

The problem is encryption can be a pain, with the constant requirement for password input to access your data. However, that doesn't need to be the case.

This article will guide you through the process of encrypting a secondary disk with LUKS (i.e. not the boot disk with your operating system installed).

Furthermore, it will detail how you can implement auto-decryption along with auto-mount, so that on system boot your data is already freely accessible. No password input!

Introduction

This guide provides information on how to encrypt a disk with LUKS, and then auto-decrypt and auto-mount the disk on boot. The following features will be included:

LUKS2 whole disk encryption
BTRFS file system, and associated settings
Auto-decryption using a keyfile and crypttab
Auto-mount using fstab

To put this guide together I have mainly referenced the Arch Wiki.

Note: the auto-decryption / auto-mounting only makes sense if the operating system is encrypted, as otherwise the keyfile used for auto-decryption would be accessible, which defeats the object of encrypting the drive in the first place.

Find target drive name

Firstly, it is necessary to find the correct hard drive. For this use the following:

lsblk -f

Example output:

NAME          FSTYPE      FSVER LABEL         UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sdf
└─sdf1        btrfs             DriveName     4177e516-e532-45e3-8ad8-81c0ec8c1696 117.2G  0%     /mnt/mount-point

In this case the drive is already in use, and has been previously partitioned.

Drive name: sdf

Partition 1 name: sdf1

Partition 1 UUID: 4177e516-e532-45e3-8ad8-81c0ec8c1696

Partition 1 Label: DriveName

Partition 1 Mountpoint: /mnt/mount-point

It may be the case that there is no partition on the disk, which is fine. The above is just for illustration.

Dismount drive if required

If necessary dismount the drive.

This can either be done in file explorer or on the commandline if the mountpoint is known:

sudo umount /mnt/mount-point

Securely wipe the drive

Photo by Rohit Gupta from Pixabay

It is a good idea to wipe the disk before proceeding any further.

Create a container called wipe_me.

Note: the "block-device" should be the main device not one of the device partitions. For example, it could be /dev/sdf , but not /dev/sdf1 , or /dev/nvme0n1 but not /dev/nvmen0n1p1.

NOTE: YOU ARE ABOUT TO WIPE THE DISK! BE SURE YOU DON'T NEED THE DATA ON THE DISK AS IT IS NOT RECOVERABLE!

sudo cryptsetup open --type plain -d /dev/urandom /dev/<block-device> wipe_me

Zero out the container. This may take a while depending on the size and type of drive:

sudo dd bs=1M if=/dev/zero of=/dev/mapper/wipe_me status=progress

Then close the container

sudo cryptsetup close wipe_me

Create LUKS encryption

Photo by cottonbro studio

Note: if you want to setup any partitions on the disk, this is the point at which you would do it. It is not necessary if you intend to use the whole disk as one single space, but it is completely up to you.

Now that the drive has been wiped the drive can be formatted with a LUKS container.

After running the below you will be prompted to input a password. This password is the encryption/decryption password and should be as strong as possible. You must remember this password as without it you will not be able to decrypt the drive in the future:

sudo cryptsetup luksFormat -h sha512 -i 5000 /dev/<block-device>

Note: I have upped the iteration parameter to 5000 from the default 3000. This will result in the unlocking of the encrypted partition taking a little longer (nothing excessive, but it depends on hardware). If this is a problem (i.e. you have a slow processor), please change the 5000 in the command above back to 3000. I would not recommend going any lower than the default of 3000.

Add Keyfile

In the previous section a password was created for the encryption. To allow the drive to be mounted on boot, without the requirement for a password every time, a keyfile will be created that will auto-decrypt the LUKS partition on boot:

Note: you can change "keyfile.key" to whatever you like, just make sure you are consistent moving forward.

sudo dd bs=512 count=4 if=/dev/random iflag=fullblock | sudo install -m 0600 /dev/stdin /etc/cryptsetup-keys.d/keyfile.key

Add the keyfile to the LUKS partition (after running the command below you will be prompted to input the encryption password you created earlier):

sudo cryptsetup luksAddKey /dev/<block-device> /etc/cryptsetup-keys.d/keyfile.key

Create filesystem

Image by joffi from Pixabay

First, open the encrypted partition (you will be prompted for the password):

Note: the word crypt is arbitrary, you can use whatever you like, just be consistent.

sudo cryptsetup luksOpen /dev/<block-device> crypt

Now create the drive file system. In this case I will use BTRFS, but it can be swapped out for whichever file system you prefer:

Note: "MyDriveName" is the drive "Label", and can be whatever you like. This is the name that will appear in your file explorer as the name of the drive.

sudo mkfs.btrfs -L MyDriveName /dev/mapper/crypt

An example output after running this command:

Label:              MyDriveName
UUID:               9f4f484c-6f5f-493f-a2a2-7c136e193e8d
Node size:          16384
Sector size:        4096	(CPU page size: 4096)
Filesystem size:    119.23GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       yes
Zoned device:       no
Features:           extref, skinny-metadata, no-holes, free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1   119.23GiB  /dev/mapper/crypt

Setup auto-decrypt and auto-mount

Photo by cottonbro studio

Essentially the drive is setup at this point, all that is left to do is setup the auto-decryption followed by auto-mounting the drive.

Auto-decryption

For auto-decryption crypttab will be used.

First, the UUID of the encrypted LUKS container is needed:

lsblk -f

Result (something like this), be sure to find the correct device as we did earlier:

NAME           FSTYPE         FSVER LABEL    UUID
sdf            crypto_LUKS 2                 3618e759-b3c7-4cf8-9753-5571ce38bc22
└─crypt btrfs                 MyDriveName    9f4f484c-6f5f-493f-a2a2-7c136e193e8d

Now add the following line to /etc/crypttab:

Note: "drive-name-crypt" is arbitrary, and use the UUID from the <block-device> line (i.e. sdf above, remember your device may not be called sdf! Pick the correct block device)

drive-name-crypt  UUID=3618e759-b3c7-4cf8-9753-5571ce38bc22  /etc/cryptsetup-keys.d/keyfile.key nofail

Note: in the above the last item is nofail. This is not essential to include, but is advisable. nofail ensures that if the decryption fails it will not cause boot to fail. You will know it has failed as the drive won't appear later on, but at least your operating system will boot!

Auto-mount with fstab

First, create the mount point if it doesn't already exist:

sudo mkdir /mnt/my-drive-mount-point

Then add the following to /etc/fstab. The UUID is from the "btrfs" line from the output of lsblk -f. I have provided two options, one for SSDs and one for hard drives (rotational drives):

SSD:

UUID=9f4f484c-6f5f-493f-a2a2-7c136e193e8d  /mnt/my-drive-mount-point btrfs rw,nofail,noatime,compress=zstd:1,ssd,discard=async,space_cache=v2,x-gvfs-show 0 0

HDD:

UUID=9f4f484c-6f5f-493f-a2a2-7c136e193e8d  /mnt/my-drive-mount-point btrfs rw,nofail,noatime,compress=zstd:1,nossd,space_cache=v2,autodefrag,x-gvfs-show 0 0

All the options after "btrfs" can in theory be changed, some explanations are given below.

Options:

ssd - specification of the allocation scheme suitable for SSD drives (rather than rotational drives). Change this to nossd for rotational drives
noatime - significantly improves read intensive workload performance, and also reduces writes in some circumstances.
compress=zstd:1 - zstd level 1 is the minimum level of compression. zstd accepts a value range from 1-15, with higher levels trading speed and memory for higher compression ratios. The default compression level is zstd level 3. zstd level 1 already gives a quite decent level of compression, increasing the compression level has a much larger impact on compression speed than it does on reducing file size, so only increase the level if you are not concerned at all with time to write files. Decompression speed is largely unchanged regardless of compression level.
space_cache=v2 - creates cache in memory for greatly improved performance. Be sure to use v2 and not v1.
discard=async - keeps the disk tidy by enabling the discarding of freed file blocks. Specifically, the asynchronous mode (async) gathers extents in larger chunks before sending them to the devices for TRIM. (only required for SSDs).

Please take a look at the official documentation for further details.

Reboot

Now reboot and if all has gone well you should see the drive appear after boot ready for use.

The only final step you may need to perform is to change ownership of the drive to your user account the first time you use it. It will likely be assigned to root the first time you boot (replace "username" with your actual username.)

sudo chown -R username:username /mnt/my-drive-mount-point

Once the above is done, it will not be required again.

thetestspecimen

Crowdfunding – What Is It and Are Sites Like Kickstarter Safe?

What is Crowdfunding? #

Where can I go to get involved? #

So how does it work, and what do I get in return? #

What are reward/pledge levels and how do they work? #

So how is this different from buying something from an online shop? #

Is it possible for me to lose my money? #

It is less convenient than an online shop, and potentially I could lose my money, so why would I bother? #

Conclusion #

Nextbit Robin Smartphone – Kickstarter Edition (Unboxing and Photos)

Nextbit Robin - Specification and Features #

Nextbit Robin - Unboxing Video and Photos #

Nextbit Robin – Software Review and 3rd Party Launchers (Nova)

...So what is the answer? #

What is the Nextbit Robin? #

How exactly is this a cloud phone? #

Photos #

Apps #

What about the native launcher? #

Can I use a different launcher such as Nova Launcher, and what happens to the cloud functions when I do? #

Anything else I should be careful of? #

In conclusion #

My Android App Development Journey (So Far), and Some Guidance for Others

Why computer programming? #

I like technical subjects #

Accessibility #

Why did you pick Android? #

My criteria #

Thinning it out #

The decision #

Where did you begin? #

Learning the basics #

Recommendation #

Practise makes perfect #

Which software did you use? #

Other useful software #

IntelliJ #

Sublime Text #

How did you progress and what were some challenges you faced? #

First Steps #

Android Studio #

Learn from others #

Progression #

I'm a coding god! #

Ok, I'm not a coding god... #

What about user login and data backup and sync? #

Getting a server #

Another recommendation #

One last hurdle #

Finishing up #

What I achieved #

Would you change anything looking back? #

What are you going to do now? #

Linux – Should I Switch from Windows to Linux (Ubuntu, Fedora) or Not?

What do I use and prefer? #

If I decide to use Linux, which distribution should I use? #

What are the advantages of using Linux? #

What are the disadvantages of using Linux? #

Who shouldn't be using Linux? #

Who should be using Linux? #

What if I need programs like Microsoft Office? #

What about the commandline, that sounds scary? #

I want to install a program #

Windows #

Fedora #

Other commands #

Conclusion #

How to Make Broadcom WiFi Modules Work in Fedora

Notes and Pre-requisites #

Let's get started #

The simple way #

Credits #

Cryptocurrency Beginners Guide – What Are They, Should I Get Some, and How Does It Work?

What I won't be talking about... #

What exactly is a cryptocurrency? #

How easy are they to use? #

The main issue currently #

What can you use cryptocurrencies for right now #

Shopping #

What is Crowdfunding?

Where can I go to get involved?

So how does it work, and what do I get in return?

What are reward/pledge levels and how do they work?

So how is this different from buying something from an online shop?

Is it possible for me to lose my money?

It is less convenient than an online shop, and potentially I could lose my money, so why would I bother?

Conclusion

Nextbit Robin - Specification and Features

Nextbit Robin - Unboxing Video and Photos

...So what is the answer?

What is the Nextbit Robin?

How exactly is this a cloud phone?

Photos

Apps

What about the native launcher?

Can I use a different launcher such as Nova Launcher, and what happens to the cloud functions when I do?

Anything else I should be careful of?

In conclusion

Why computer programming?

I like technical subjects

Accessibility

Why did you pick Android?

My criteria

Thinning it out

The decision

Where did you begin?

Learning the basics

Recommendation

Practise makes perfect

Which software did you use?

Other useful software

IntelliJ

Sublime Text

How did you progress and what were some challenges you faced?

First Steps

Android Studio

Learn from others

Progression

I'm a coding god!

Ok, I'm not a coding god...

What about user login and data backup and sync?

Getting a server

Another recommendation

One last hurdle

Finishing up

What I achieved

Would you change anything looking back?

What are you going to do now?

What do I use and prefer?

If I decide to use Linux, which distribution should I use?

What are the advantages of using Linux?

What are the disadvantages of using Linux?

Who shouldn't be using Linux?

Who should be using Linux?

What if I need programs like Microsoft Office?

What about the commandline, that sounds scary?

I want to install a program

Windows

Fedora

Other commands

Conclusion

Notes and Pre-requisites

Let's get started

The simple way

Credits

What I won't be talking about...

What exactly is a cryptocurrency?

How easy are they to use?

The main issue currently

What can you use cryptocurrencies for right now

Shopping

International money transfer

It will improve

What do all the new words mean?

Blockchain

Private Key

Public Key

Wallet

Wallet Seed or Seed Words