Category: Technology

Syncthing

I seem to be constantly trying out different ways of syncing data. I have had no end of problems with unison. Apart from being slow getting build versions to match is a constant source of frustration.

However,  I have found Syncthing, which seems great so far. I have owncloud, which is great for syncing documents; think Dropbox that you own. However I have some data sets that are very large and don’t change much. Owncloud would just get clogged up looking after these. I could use rsync, which is fine, however syncthing has some nice additional features.

Syncthing is a bidirectional sync tool, that can also do versioning if you want it to. It works by running as a web service on your computers and is accessible via a web gui. It keeps directories up to date over a number of computers, there is no copy of the files in the cloud, unless you put one there. So if you delete everything off one computer it will disappear from them all (eventually). It plays well with dynamic IP addresses and firewalls. All traffic between the computers is encrypted, and it is fast. It doesn’t use up too much resource, and I set the sync frequency on my files for 1hr as they are unlikely to change that often, and also only I use them. These are large data sets of source material mainly, and photos that I edit on two computers.

Setup is pretty straightforward, the most tricky bit being auto starting the service on different computers/operating systems. It works well.

OwnCloud – its awesome!

I gave up on SpiderOak in the end, it was too resource heavy for my laptops, their batteries were really struggling. The constant encrypting and decrypting was too much. Its a shame because as a backup system for large amounts of data it was really very good. Especially if that data was on desktops or didn’t change much. I couldn’t justify keeping it for this use only, too expensive at $25 a month (which is about £20 now!). So I looked back into OwnCloud.

I tried OwnCloud a number of years ago as I run my own home server anyway. Back then the desktop sync client was just not up to the job. It would take forever to sort out changes and sync them, often using up entire CPU cores on both the server and the client machines. I am pleased to say that problem as been solved and now OwnCloud is a great choice if you are running your own server somewhere anyway. The sync seems to be fast, and the setup is easy. I am using a self-signed SSL certificate, which doesn’t cause much in the way of annoying warnings. OwnCloud has the other obvious advantage that you can really limit the access to the data, and it supports encryption of the data on the server too. Or you can encrypt the drives etc. I have been using it for about 4 months without problem. The only issue I had was when there was a power cut in my house while I was away. Doh! This killed the server obviously.

The downside of course is that with SpiderOak I had a ‘cloud’ copy of all my important data that was at a different location to my house. This is no longer the case. I don’t use OwnCloud to sync all the large data-sets across my computers, I sync those with Unison instead. So to avoid a catastrophic data loss in the event of disaster I have synced all my important data and docs to my office computer using Unison. This is about 1.7TB of data onto a 5TB raid array. The first sync took ages but as this data doesn’t change very much. I also use Amazon Glacier for data that can be essentially archived off and forgotten about, its super cheap.

Let’s Encrypt!

I had to do it eventually. My security expert credentials (such as they are) were coming under question from my lack of https on my web server. This was partly due to that until recently it would have required additional expense, as additional IPs would been needed and also SSL certificates. Not so anymore! Let’s Encrypt is a great service where you can get free SSL certificates for your website, although I highly recommend giving a small donation if you are able. At this point I would normally write a blog post detailing how I setup the service for my server and websites. However as I run my server on Bytemark’s Symbiosis I didn’t have to do anything other than turn it on. All config done automatically, I did add a file named ‘ssl-only’ to the ‘config’ dir of my hosts to ensure that all traffic is redirected to https. I would have done it sooner but I needed to upgrade the OS on my server and in order to do that I needed to check backups and all that stuff that you should do anyway.

Chromebook copy-paste from PDF file

This is the first time I have travelled for an extended period of time with only my Chromebook. I would normally take a laptop if I was going to be away for more than a couple of days. Not only that I have to write and then present a talk at a conference. So I need Google Slides to do the business. I was reasonably confident it would, there is however one snag.

I have had bit of a faff copying some sections out of a PDF file to go into my presentation. There is no way to easily copy and paste from any of the PDF tools I have used and Slides will not import a PDF file as an image. This is either because the PDF support in Google Docs just isn’t very mature or a priority for Google or, and I have my suspicion this is part of it, Google doesn’t want you copying out of PDF files. I am not entirely sure why this might be, perhaps they are trying to protect copyright? However, it can be a pain if you are trying to use a figure as part of a talk.

I have found what is a fairly good solution, in fact it might be enough of an obvious (when you figure it out) solution that there isn’t a problem here after all. I learnt how to take screen shots with the Chromebook. Either by:
ctrl+’task select’ and you get the whole desktop, or
ctrl+shift+’task select’ and you can draw a box round a section

Easy, you can either copy the image to the clipboard or save it as a file. Probably easy enough that my conspiratorial ideas of Google somehow blocking it do not really make sense. Just took me a while to figure it out. 🙂 You can even zoom into the bit of the PDF you want before grabbing the screenshot.

A Beast

I have been cleaning up and resurrecting a beast of a computer. Its a dual Xeon 3.16GHz, with 8GB of RAM. Its not that high spec compared with today’s hardware. Has a fairly average GPU and a RAID card with currently 7x 500GB HDs, and 1TB system disk. I think I will get it going. Maybe find a use for it. Perhaps as web scraper or something. That or I could sell it. Gonna have to get a HD off eBay to make up the 8, that and find the Sata PSU cables!

Oldish Computer
Beast.

Got Me a Server Rack

I have been going through a process of tidying up! So I have got a server rack and put a load (3) of my computers in it. I got it free, thanks York University, and all it needed was a new wheel. Its working out pretty well. the Top one is my home server, the middle a Windows machine, and the bottom is my quad Opteron system that doesn’t get used much these days. Bit of a relic from the PhD days of ploughing though lots of simulations.

20161118_124912

SpiderOak and the Tail of the Dead Battery

I switched from Dropbox to SpiderOak a while ago. For a couple of reasons. One, I wanted to upload more things to the cloud for backup, video projects etc, and SpiderOak have a personal 5TB plan. I was also starting to get nervous about security and I think the encryption system in place by SpiderOak is better than Dropbox. However, this transition has its downsides.

The SpiderOak client is not as slick as Dropbox, in fact it looks rather retro. I don’t care about the retro, but the interface takes a bit of getting used to. That said, once you get your head round it it is not difficult to use. The main downside is that if you change a lot of files on SpiderOak then your computer is going to be spending a lot of time encrypting and decrypting files. Uploading a load of photographs downloaded from my digital camera took a fair while on my MacBook Pro, and that is a fair while of one core at 100% and the fan spinning away like crazy.

That is not the only way you can flatten a laptop battery very quickly. While being a Dropbox user I got into the habit of just working straight out of Dropbox. So I would incrementally save files into the Dropbox folder, and Dropbox would dutifully upload this file into the cloud. The situation with SpiderOak is somewhat different. The save event has to be detected, and then the file is encrypted into some sort of bundle of files. This is then uploaded to the cloud. I have just finished working on a Word document and when I looked at the upload queue in the SpiderOak client, three versions of the file where there waiting to go. I have also been working on my laptop for maybe 3 hrs from full charge. The relatively large amount of work that SpiderOak is doing to do all that encryption means I have warm laptop with 28% battery… not so good.

I think this system also causes problems for SpiderOak. The client reports that I am using 1.6TB of my 5TB of space. However, when the client reports the size of my files on their severs, its over 5TB. This is because of all those duplicate encrypted files. I am not sure how they go about solving that problem. The big duplications were caused when I changed round some computer systems and the client believed that the files on the hard disks were all new version of the entire content I store on the cloud. Oh dear.

So what to do. I like the extra storage space, and I like the encryption. I don’t like the battery usage or the CPU hammering. Its fine on my huge desktops that have 8+ cores and tons on ram. I don’t even notice. At the very least, live files like that Word doc, might have to be edited outside of SpiderOak and then copied in. That or I go looking for another solution.

DriverTuner and the generic solution that doesn’t work

I don’t spend much time with Windows, my desktops are largely Linux and my laptops Macs. I do have one rather ageing Windows machine, that recently got borked by an update. It was when I was reinstalling the drivers and software that Windows update couldn’t find that I came across DriverTuner. What a hateful piece of software this is. I have been using computers for years, and I started early enough to remember the days when you had to manually set IRQs on hardware with little jumpers. So tracking through a website using the model numbers of my hardware isn’t much of a hardship. I don’t believe it would be to anyone really, we can all navigate websites…

Therefore I found it extremely irritating that when trying to install both my Canon printer and Epson scanner that I kept navigating to a specific page only to download DriverTuner. On the Epson site it even downloaded in a zip file that looked like the correct version of Epson Scan. DriverTuner is irritating because it doesn’t seem to work, it failed to find anything to install on my computer. Not only that it also seemed to want me to buy it. Would it have made me buy it to get my drivers? To me DriverTuner is nothing more than an annoying, overly complex, failed, solution to a almost non-existent problem. Another piece of crapware clogging up my system.

Canon at least had a version of the drivers and software somewhere on their site that I was eventually able to locate and download. It installed and worked. Epson, not so. I had to resort to chatting to a helper to get a link to a page that I could download the driver from. Why, manufactures, insist on making everyone download this software? Why not let people try the automatic method if they want, and let those of us who want to just go to site and get the driver without any fuss do that?

Windows 10 Nvidia nForce RAID Update Troubles

So a resent update caused no end of problems with my Windows 10 machine. It would repeatedly fail to install the update, and then when it managed it I couldn’t log into the machine. User profile not accessible. I decided that it was probably time to re-install my machine anyway. However it seems the problem came back. A little different this time.

I reinstalled but chose to leave my Nvidia Soft/Fake RAID partition unchanged as I used it as my user space. The machine re-installed, and everything appeared to be fine. I could get to the RAID array, so I left it as is and went about installing drivers and updates. Updates! This is the problem. It seems I have become one of the many victims of a Windows 10 update that stops Nvidia RAID arrays from working (see here in German). So no sooner had I updated the machine and restarted, the RAID array vanished again. So annoying. This machine is a little old but it is by no means obsolete. Its a dual quad-core opteron with 16GB of ram and a half decent GPU. Plenty of live left in it yet, esp as all I use it for is scanning and printing photographs. The RAID array still reports as healthy at boot. I suppose I should have known not to trust a fake/soft RAID long term. However, wasn’t expecting an update to do it in. So what to do… Buy a cheap RAID card off ebay and use that, turn off Nvidia RAID and use the Windows fake RAID instead? Whatever, they should have warned us of this potential if they knew.

Gigabyte X79-UP4 Linux Horror

So I am not much of a Windows user, I have one Windows machine that doesn’t get used all the much. I tend to use Linux and have done for about 15 years. One of my work machines has caused a bit of trouble. The system was built around the Gigabyte X79-UP4 motherboard. This is on the face of it a good main-board, nice spec, good components. However I have had a lot of trouble with it. Its all about the USB 3 chipset, Linux hates it. The symptoms are as follows:

  • Anything plugged into the USB 3.0 ports and the machine will not shutdown, it instantly reboots.
  • USB 3.0 devices are unstable, and don’t always function.
  • With anything in the USB 3.0 ports the system can become unstable/unresponsive.

The solution was to deactivate all the USB 3.0 chipsets on the main-board. The the system behaves itself. I didn’t however discover this very quickly. I first tried replacing the PSU as I thought it had become damaged somehow. I then complained to Gigabyte who asked me to send the board back to them. Which I did at cost to me. They checked the board and said it was fine. They however refused to check the board with any Linux install. It is hard to replace a X79 main-board now. I so I have to continue to use it. I have however installed a PCIe USB 3.0 card that is now running the front ports on my system, and 2 on the back. I tried this board with a couple of difference Linux versions, including Ubuntu 14 and Opensuse 13.3/Leap 42.1. So, this means that I will probably never risk a Gigabyte project again, to risky.