Computer Gaming Technology for 2018

If you are currently or plan on playing a competitive game there are some tips and tricks to gain an advantage over your opponents, or at least put you on a level playing field. This article is broken down into the following sections:

  • Mouse
  • Keyboard
  • Headphones
  • Display
  • Computer
  • Internet

Mouse

This is probably one of the most significant factors. Almost all games including FPS, MOBA, RTSG, Battle Arena, etc. all rely on mouse movement and button presses.  We will break the mouse down into a few different sections.

DPI

Dots Per Inch; this is the speed at which your mouse moves across the screen. If your mouse does not support setting the DPI, its probably time for an upgrade. The slower your mouse moves the easier it is to focus on your target and click on heads.

FPS Recommended DPI: 400

If you are throwing down with all the new Battle Royale’s coming out this year you will want to drop your DPI down to 400 and start working your arm. This is how all the professional gamers play. Very few professionals play at 800 or higher.

Mouse Pad

A gaming mouse pad is extremely important for tracking and control. You want to get the Medium/Large (12″x14″) size pad. If you have room get the Large/Extended size. At 400 DPI you will be picking up your mouse quite a bit.

The ideal mouse pad is a cloth pad optimized for control/precision.

I would recommend one of these:

Gaming Mouse

If you don’t have a gaming mouse you should get one. Key features of gaming mice include the CMOS sensor DPI range, buttons/layout and software for customization of buttons and functionality.

I use a gaming mouse for every day use. Their superior sensors provide almost perfect tracking and I rebind all the extra buttons to extra functionality. I also require 3600 DPI for programming which is not available in cheaper mice.

Logitech G502 Proteus Spectrum

https://www.logitechg.com/en-us/product/g502-proteus-spectrum-rgb-gaming-mouse

This is currently regarded as the worlds best gaming mouse. I also use this mouse. While the mouse could see many improvements it has 1 major flaw; the middle mouse button takes too much force to activate and usually causes an accidental side-scroll click. The solution is to re-bind middle click to one of the other mouse buttons. The mouse feels way too good in my hand and has too many extra buttons to throw it out over that. The Logitech mechanical switches are a little clicky too.

Keyboard

This won’t help you as much as a mouse, but once you go mechanical you never go back to rubber. I use a WASD code but any keyboard with your preferred switches will work.

Don’t get suckered into Cherry MX Blue or any audible keys. You want to listen to your game and not your keystrokes. I use Cherry MX Brown because they provide tactile feedback when pressed that you can feel.

http://www.wasdkeyboards.com/index.php/products/code-keyboard/code-87-key-mechanical-keyboard-2563.html

The WASD Code may not be for you. Make sure you get a 10-keyless (87-key) mechanical keyboard. Getting a 10-keyless is very important. You will need the extra space to move your mouse. You can always buy a separate keypad if you use that when not gaming.

Gamepad

Keyboards are not designed for gaming. The thumb sits idle the majority of the time only hitting Space and Alt for most. A gamepad can put the thumb to work and provide much more functionality. My main problem with gamepads right now is they take away too much functionality from the main hand, most importantly the pointer finger. The pointer finger is not used for WASD and most gamepads take away the second row BGT5 which I use heavily. If I can find a gamepad that makes good use of the thumb and has enough keyboard functionality, I will make the purchase and give it a try.

If you don’t use the extra row for the pointer finger and want to do more with your thumb a gamepad might be right for you. Again, don’t go cheap, make sure you get one with good mechanical switches. Take a look at the Razer Orbweaver Chrome:

https://www.razer.com/gaming-keyboards-keypads/razer-orbweaver-chroma

Headphones

A good pair of headphones is important to gaming success and there is a lot to learn about headphones; some of which I need to cover so I will try to keep this brief. But lets be clear that we are talking about headphones, not a gaming headset; avoid those.

Open Back

Open headphones are exactly that, open. They let sound in, and out. You can hear things around you and things around you can hear whats playing on your headphones.

The drawback is people around you can hear everything that is playing on your headphones. If that is an issue, you will have to get closed back headphones.

Benefits include better sound quality and a better sound stage. This will allow you better hear directional sound. You can also hear your self talk, which is very useful for communications.

Closed Back

By now you can guess these are closed headphones. By default they usually include noise isolation which prevents noise from escaping your headphones. This benefit allows you to listen to your headphones anywhere, work, school, etc. with out disturbing the people around you.

The drawbacks to this is the sound stage will be much smaller and some directional audio may be harder to dial in on. Also you may not be able to hear your self speak which can effect communications.

Amplifier

After using a headphone amplifier I can’t imagine listening to headphones without one. Many high end headphones require an amplifier. Explaining amplifiers will take as long as explaining headphones; so just know if your headphones need one or not. If you are on the fence of getting an amp I strongly recommend it. Even some cheap headphones can sound better with a little power. If you are new to the game, I recommend the Schiit (yes, pronounced “shit”) Magni 3.

http://www.schiit.com/products/magni-3

Conclusion

I am no audiophile and already spent $400 on my Sennheiser PXC 550’s for work, so went cheap for my gaming headphones and got an entry level pair. For sub $200 the Beyerdynamic DT 990 PRO’s don’t have the best low’s when listening to music, but have well defined mid’s and high’s and you won’t miss a thing while gaming. Be warned these are studio grade headphones and require an amplifier to work; they will sound like a trash can with out one.

https://north-america.beyerdynamic.com/catalog/product/view/id/352/s/dt-990-pro/category/1464/

There are many alternatives and many better headphones, but not a single pair is cheaper. A good pair of headphones will cost about $300-$500.

Display

Unless you have a 120Hz, 144Hz or faster monitor, you probably have a 60Hz monitor. This means your monitor can display a maximum of 60 FPS. Even if your computer is going at 240 FPS your monitor is still limiting you to 60 FPS. Once the rest of your equipment is up to speed you may start to notice your slow monitor. Trying to click on a moving target at high speeds can be hard when your monitor is only refreshing 60 times a second.

Another important feature of a new monitor is G-Sync (requires compatible nVidia GPU). Now that you know your monitor is refreshing X times a second (based on the speed in Hz) this may not be 1:1 for the FPS your GPU is able to produce. G-Sync is a module in your GPU and your Monitor that dynamically changes the refresh rate of your monitor to match the FPS of your game so you always get a 1:1 refresh.

That being said, let go over the key features of a gaming monitor.

  • 24 or 25 Inches
  • 1080p (1920×1080)
  • 1 ms Response Time
  • 144Hz or Faster Refresh Rate
  • G-Sync

And the winner is the Dell Alienware 25 Gaming Monitor. If you watch carefully you can some times find it on sale for $379.99.

https://www.dell.com/en-us/shop/alienware-25-gaming-monitor-aw2518h/apd/210-amsr/monitor-accessories

Computer

While you don’t need a “Gaming Computer” you will need a rig with enough power to drive your games. This is not a computer gaming article but I will cover the basics.

GPU

G-Sync has been supported since the GTX 660. If you can afford it get the GTX 1080 Ti. You basically want the most powerful single GPU you can buy. Avoid SLI until you have a Titan Xp (or two).

Note: I would avoid buying a GPU this year if you can due to cryptocurrency.

CPU

Yes, the faster the better. Any newer generation i5 or i7 should work just fine. I still use a second generation i7-3770k and run most games just fine. Avoid AMD and stick to Intel.

RAM

Least important specification but you do want enough memory to run everything and have plenty to spare. I recommend at least 16 GB. Faster RAM is better but you wont see a huge performance increase.

Internet

While there may not be much you can do about this for a reasonable price it is worth mentioning. Even with the best hardware in the world if you have a bad connection you will never hit your shots. You want to get the fastest internet possible; not to be confused with bandwidth. Speed is latency (ping, ms). This is how long it takes you to send bits of data from your computer to the game server and back. The faster you can do this, the more advantage you will have over opponents. Short of moving there is not much you can do other than making sure you have the fastest ISP in your area.

That being said if you are willing to pay a monthly fee you may be able to find a service to improve your connection to your game servers; depending on what game you play and where you live. For example, lets say it takes 16 hops (network devices between you and the game server) for you to reach your game server using your ISP’s route. An ISP like Akamai usually has end points within 1 or 2 hops of the majority of the populated world that uses the internet as well as pipes around the world. This allows them to offer routing services where you connect to them, take their pipe to the game server location, then one last hop to the game server. That would reduce your total hops from 16 to about 5 or 6 as well as take their faster route substantially reducing your latency. Now don’t get excited because unless you can find existing services for the game you play that work for you, a custom request would cost thousands of dollars to configure plus service and maintenance fees. I used one of these services for World of Warcraft back when I played competitively and it reduced my latency by about 40%.

Automate Manual Payments with Bill Pay

In once sentence, automate your rent and other checks with your banks Bill Pay for free.

These days almost everything allows you to configure automatic payments. There are still a few occasions when you need to write a check. In many cases landlords and other services still only accept check payments. Most major banks provide a free Bill Pay service that can be used to manually or automatically send checks to recipients with custom memos.

Overview

Bill Pay can allow you to do the following:

  • Automatically send checks to designated payees.
  • Send them manually or on a recurring schedule.
  • Pay other bills, get notifications, reminders and alerts.

Getting Started

First you need to make sure your bank supports Bill Pay. You log into your banks portal and look for a Bill Pay section. For a list of banks that support Bill Pay see this article:

https://www.nerdwallet.com/blog/banking/online-bill-pay-what-it-is-why-you-should-use-it

The steps will be somewhat different based on your bank but the process is about the same and very simple.

  1. Setup Bill Pay at your banks website.
  2. Add payee’s you wish to send payments too.
  3. Send a one-time payment or setup recurring payments.
  4. Set an amount and custom memo to appear on the check.
  5. Choose a send-on and receive-by date for the check to be sent.

Your bank will mail a check to the payee’s address for the amount you specify. You may be able to configure alerts and notifications for the payments.

Conclusion

Stop writing checks for rent and other recurring services. If your bank does not support Bill Pay I suggest you look for a good one that does. While I do not recommend one of the Big Four you can get $200 to open a Chase checking account.

Zelle – A Faster Built-In Alternative to Square and Venmo

Zelle is an app to send money to family and friends you know directly from your banking app, instantly and free.

https://www.zellepay.com/get-started

Overview

Why should you use Zelle?

  • Built-In to your banking app, nothing to download or signup for.
  • Send money instantly from your bank to theirs.
  • Completely free, no costs at all.

What are the downsides?

  • Not all banks support Zelle (most major banks do).
  • Sender’s full name is disclosed to the recipient.

Zelle does have an app that works with your debit card if your bank is not supported. I have not had the ability to test this due to all my banks supporting Zelle. Only use this if your banking app does not support Zelle:

https://www.zellepay.com/go/zelle

Details

So why is Zelle able to provide instant, free service unlike competitors (Venmo and Square); who charge a fee for instant transfers? The short answer, Zelle is owned by the banks.

Competitors are separate businesses from banks and need to make money also. If they allowed you to instantly transfer the funds, they would not be able to hold on to the funds and earn interest on it like banks do. This is why you must wait at least one day or pay a 1% fee to instantly receive the funds.

During that one day the funds are sitting in the competitors accounts, banks are not making money on those funds. Zelle is owned by the banks and offers free instant transfers to combat the competitors. By not transferring the funds to the competitors for a day the banks continue to earn the interest on the funds.

Getting Started

First check to see if your banking app supports Zelle.

https://www.zellepay.com/get-started

If so, you should fine more info how to Enroll your mobile number or e-mail address to receive Zelle payments. Enrollment lets you choose your pay from account and pay to addresses. The recipients for which you wish to send money too must also Enroll with Zelle.

Once both parties are enrolled with Zelle and you have their confirmed phone number or e-mail address, you should be able to send them money instantly from your phone app or banking website.

For more information, visit Zelle at https://www.zellepay.com/

Best Financial Apps and Tools for 2018

From sending money, to tracking expenses, to trading stocks and options; these are some of the best financial tools available today.

In no particular order, we now present…

Personal Capital

https://share.personalcapital.com/x/50KL83

This is probably #1 (ignoring the order above comment) since it is able to track almost every aspect of your finances. This app will automatically track your expenses, income, investments, net worth, and more. There is a full article here:

https://shane.pw/2018/05/14/track-spending-income-assets-investments-with-personal-capital/

Robinhood

Free Stock:  https://share.robinhood.com/shaner195

Robinhood, while not the best broker available, is my top recommendation and primary app for trading stocks and options. Enter the new world of zero trade fees on all stock and options trades. You even get 1 free stock just for signing up here. I have been using Robinhood all year and not looking back. There is a full article here:

https://shane.pw/2018/02/14/free-stock-trading-with-robinhood/

Cash App

Free $5: https://cash.me/app/BBWCDRD

Square is an amazing financial services company and their Cash App is just as amazing. This app allows you to send and receive money at no charge to friends, family or co-workers. Great for paying back lunch, splitting bills, or sending someone money. Square offers many other financial services as well. I have used it side by side with Venmo and in many instances Venmo has been up to 1 day slower to deposit funds in my bank, and up to 5 days slower to charge my card for funds sent; I also do not like supporting PayPal. Get $5 free after you send $5 if you sign up here.

Update: An alternative to Square that is instant, free and built-in to your banking app is Zelle. Use it to send money to your friends and family:

https://shane.pw/2018/05/19/zelle-a-faster-built-in-alternative-to-square-and-venmo/

Credit Karma

https://www.creditkarma.com/

Probably the best credit monitoring/tracking app available. You get weekly updates from TransUnion and Equifax. The app also presents your accounts and the details in a very easy to view layout. The score is VantageScore 3 but the goal here is to monitor what is on your credit, and not the number. For more information on monitoring your credit and credit scores see:

https://shane.pw/2018/04/27/credit-score-information-monitoring-and-its-importance/

Simple

https://www.simple.com/

Simply put (pun intended), Simple is a bank inside an app. If you don’t want a brick and mortar bank and want to do it all from your phone then Simple is for you. Turn most ATM’s into your local branch for withdrawing cash. Transfer money to and from accounts, including external accounts at no charge. If you already have a bank and it is working for you, then no reason to switch. I used Simple until they were purchased by BBVA Compass at which time I let the account go. I really have nothing bad to say about them and may start using them again in the future.

Dollar Bird

https://dollarbird.co/

Many of you won’t find a need for this app. I like it for its simplicity. The primary use for this app is to monitor the balance in my primary checking account. Not many transactions go through my bank since I run everything possible on a credit card for cash back. I am able to setup recurring payments like rent, loans and auto-pay credit cards, plus recurring income like salary. Everything else is super easy to manually enter like credit card payments, transfers, etc. You can also change the date or amount on a recurring payment with or without effecting the series. The only downfall is it shows your current balance in the primary display area and does not calculate future debits before credits (total before next pay day). You can however see the available balance for any day. To help visualize my balances I set my auto-debits for the pay period to deduct on the same day I get paid so the totals are more accurate.

Your Banking or Credit App

Regardless of your bank or credit card you should have your banks app downloaded on your phone. Most major banks offer useful features that usually include:

  • Mobile check deposits.
  • Deposit and spending notifications.
  • View, transfer, pay, and manage your funds.

You can usually download an app for each of your banks and credit cards. For me the most valuable feature is the notifications. I am instantly alerted any time a transaction is made that meets predefined criteria.

Track Spending, Income, Assets and Investments with Personal Capital

In once sentence, Personal Capital is an app to automatically track all your spending, income, assets, investments and net worth.

Get $20 Amazon Gift Card here: https://share.personalcapital.com/x/1GnotA

Overview

While Personal Capital offers premium investment management features for portfolios over $100,000, I am more interested in the free expense, income, asset, spending, and portfolio tracking features.

Pros

  • Ability to link all your accounts (US Only).
  • Track all your spending/income by category and date.
  • Track all your portfolios change and distribution.
  • Add manual accounts and portfolios (tickers auto update).
  • Beta support for cryptocurrency (plus Chrome Extension support).
  • Automatic Net Worth calculations (assets and liabilities).
  • Real estate valuation through Zillow (if you count this).

Cons

  • No API (as a programmer this is huge downside).
  • Expense categories are sometimes inaccurate (easy to fix).
  • Cryptocurrency only seems manageable through Chrome extensions.
  • Unable to export reports.

Key Features

In the short time I have been using the app, these are some of the feature that I like the most.

  • Ability to view all your expenses across all accounts by time and date.
  • Compare income, expense, savings, and investments side by side.
  • Excellent Net Worth calculator, all done automatically.
  • Track all your investments individually or as a whole.

Getting Started

They require a mobile phone for two-factor security, otherwise, just head over to Personal Capital and sign-up.

The first think you want to do is add all your credit cards. This will allow you to track your spending by category and date. You should also link your bank accounts to track loan and mortgage payments, or other expenses that go through your bank. You will want to go through and check the categories to make sure their correct. You may also want to create some custom categories to help you better track your spending.

Second, you can add the rest of your assets including your investing accounts, Robinhood, Ameritrade, and retirement funds like 401(k)’s and IRA’s. This will allow you to track your growth and holdings of your investments. You can view individual or all accounts, the holdings breakdown, and return on investments.

Lastly, if you have any accounts that are not supported or manually managed, you can add many different manual accounts. Manual investing accounts with matching tickers will automatically update. There is a Chrome extension called pfcrypto that will update your custom cryptocurrency holdings every time you log in.

Tips and Tricks

Expense and Income Categories

Not only were some of these incorrect, I also wanted to place some expenses in custom categories. You can easily search and edit multiple transactions at the same time.

External Accounts

I got an error adding my Robinhood account, but it eventually showed up with all the information. I was unable to add my Acorns account, it keeps saying  bad username and password (I don’t plan on using Acorns anyway). And I had to manually add my cryptocurrency holdings and keep them up to date with a chrome extension.

Security

The first thing I checked into. They use Yodlee and all your access credentials are stored with them. The web interface is protected with two factor authentication. You can not transfer money between accounts or make changes from Personal Capital.

https://www.personalcapital.com/financial-software/security

For a full security review, check out this article:

http://www.bayalisistheanswer.com/personal-capital-security-explained-part1/

Conclusion

I strongly recommend this app to anyone looking to track expenses or net worth. I am still new to this app but extremely satisfied with the features available. I have a very strong hope that an API becomes available in the future. Let me say this again, I really want to see an API!

I will continue evaluating and make updates if any breaking information comes to light. Feel free to leave comments if you have anything additional to add that I have left out.

Sign Up here: https://share.personalcapital.com/x/1GnotA

Add Fingerprint Login to Your Windows Desktop for Under $50

For a while I was in search of a biometric fingerprint reader to use with my desktop which runs Windows 10. This function was primary needed at work where I frequently lock and unlock my PC but has been very useful at home too. All the desktop fingerprint readers were over priced and lacked quality.

I eventually gave in and tested a USB fingerprint reader and USB extension cable. The read was almost instant, worked almost every time at many different angles. After 6 months of use I am still very please and have recommended this to many satisfied friends.

The Setup

I use the USB fingerprint reader with a desktop USB extension cable. The total cost for me with Amazon Prime including tax was $41.41 .

At Home

At Work

The base is very sturdy and holds the fingerprint reader straight up.

Fingerprint Reader – $29.99

https://www.amazon.com/gp/product/B06XG4MHFJ

USB Extension Cable – $7.99

https://www.amazon.com/gp/product/B001K9BFB8

More Details

Tested working with:

  • Windows 10 Professional
  • Windows 10 Enterprise

Note: Enterprise requires administrative access to enable PIN login.

No third party drivers were required.

PS: If you have tested this with any other software configuration please post your results in the comments below. Thanks.

New PC Build 2017

Overview

I have not yet purchased this computer, but plan to do so for the holidays.

The inspiration for the new PC is the huge performance boost provided by the Intel X-Series processors.

Full Parts List

Newegg: https://secure.newegg.com/Wishlist/PublicWishlistDetail?WishListNumber=3245895

Amazon: http://a.co/gKGHp1n

Parts Details

Processor

With the arrival of the X-Series processors from Intel, its very clear the best bang for your buck is the i7-7820X. This provides ample performance and can be squeezed into a sub $2K PC.

Motherboard

The ASUS ROG STRIX X299-E is a pretty solid FCLGA2066 socket motherboard with all the connections I need. There is a Deluxe version but its  a bit more pricey.

RAM

G.SKILL TridentZ Series is slightly taller than traditional G.SKILL dimms, but this is 3200 MHz RAM makes use of the extra large heat spreader. I chose the white/black to match the case.

SSD

This includes the latest M.2 SSD slot with the Samsung 960 EVO. While there is a Pro version available, the performance is only slightly better and not worth the extra dollar.

Case

I chose the proven Fractal Design Define R5 due to its quiet 140mm fans all around with plenty of room for a large cooler. There are many other case options available with fully customizable designs and support for things like RGB.

Cooling

It includes the air cooler due to providing better cooling performance and lower noise output compared to water coolers. Water cooling looks cool, but does not cool as well, and makes more noise.

Power

The Seasonic PRIME TITANIUM 750W PSU is more than enough to power what will be going in this PC.

Other Peripherals

GPU

I don’t plan on getting a new GPU with the computer. Get the best Nvidia GPU you can afford. Never go SLI until you have the most powerful single card available. For a good  balance between budget and performance, I recommend the EVGA GTX 1080 FTW:

https://www.amazon.com/dp/B01GAI64GO

Monitor

A Monitor is very important to gaming and these are the qualities you want to look for.

  • G-SYNC – This will keep games smooth when your FPS drops below your monitors refresh rate.
  • 144 Hz Refresh Rate – You need at least a 144Hz refresh rate, anything higher is only marginally better.
  • Low Response Time – Generally you want a TN panel for 1 ms response times. With an IPS panel, you can get 4ms response times.
  • 1080p Resolution – If you get a 1440p monitor your FPS will take a huge hit with over 60% more pixels to fill.

The monitor I plan on ordering is the ASUS ROG SWIFT PG248Q with a 1ms response time TN panel and 180Hz refresh rate:

24″ TN 180Hz: https://www.amazon.com/dp/B01M4II4M4

If you think you need a 240Hz refresh rate, there is a 240Hz version for about $100 more:

24.5″ TN 240Hz: https://www.amazon.com/dp/B06WRP3F5S

Lastly, if you want the 1440p IPS experience this is probably the best monitor on the market:

27″ IPS 165Hz: https://www.amazon.com/dp/B017EVR2VM

Intel Announces 8th Gen Processors: Coffee Lake

If you cant afford the Intel X-Series, the next generation of Intel mainstream processors will be dropping this year. Coffee Lake looks quite impressive with 6 cores and 12 threads on the i7.

These puppies run on the Z370 chipset and will require a motherboard upgrade to use.

https://ark.intel.com/products/125903/Intel-Z370-Chipset

Not much to say until these actually drop and the benchmarks are in; however, I have a feeling they will be quite impressive compared to the previous generation. You can see the full specs of the flagship i7-8700K here:

https://ark.intel.com/products/126684/Intel-Core-i7-8700K-Processor-12M-Cache-up-to-4_70-GHz

You can read more about this lineup here:

https://www.anandtech.com/show/11869/intel-announces-8th-generation-coffee-lake-hex-core-desktop-processors

 

Vultr: Digital Ocean Cloud w/ 50% Savings

Vultr is now providing a “Digital Ocean” quality control panel and API with dozens of locations for half the price.

Compared to Digital Ocean the $5/mo offering has 2x the RAM (1GB vs 512MB), 25 GB of storage (compared to 30GB) and both have 1 vCore.

Notable features of Vultr include the ability to easily add and use startup scripts, uploading of custom ISO’s and – currently – free snapshots. All servers are billed hourly and can be created and destroyed on demand without  commitment.

Vultr uses KVM, Intel processors, SSD storage, and 1 Gbps network. All standard features are included such as automatic backups, local networking, ipv6, and – for an extra cost – fail over IP addresses.

It’s not disclosed, but the $2.50/mo offering is limited to a maximum of 2 instances. This is only a drawback when seeking a large number of micro nodes.

I have been using Vultr for over a year and very pleased with their service. I am even more excited with the new price range providing me with 50% savings. Vultr is a great choice for both new-comers to the cloud as well as veteran cloud automagitions.

GitLab Restore of git.cssnr.com

Late Saturday night, early Sunday morning, I was completing routine maintenance/upgrades which regularly included GitLab. Previously I had made some changes to the GitLab server, cutting back worker threads in an attempt to reduce memory usage. The changes included cutting the server down to 2.5 GB of memory from 3.5 GB and adding 1 GB of swap space. Things had been running fine, but I could tell by the graphs that it was still struggling.

Before the upgrade, I noticed GitLab was using 100% of swap space, and all of its memory with very little in cache/buffers. Therefore, I decided to shut it down to increase the memory to 3 GB. Randomly before shutdown I decided take a manual GitLab backup; something usually only done before performing the upgrade. After increasing the memory and attempting to boot the server, the server instantly failed to boot.  The error messages made the problem plain as day to me.

Unable to locate image /qcow2_sda/git01.1702052226.qcow2

Images with this naming convention are created by backup automation I have in place. In order to back up a VM, you must take an external snapshot creating a new delta image that all new changes are written to, locking the original base image as a read-only to be referred to by the delta(s). While new changes are being written to the delta image, you can safely back up the read-only base image. Once backup operations are complete, you must block-commit the delta image back to the base image and pivot back to that disk. If this process fails, you are left with the VM still writing new changes to the delta image, and no change being written to the read-only base image.

After much confusion, I eventually determining the delta image GitLab was running on did not exist and was cleaned up weeks ago. Therefore, I proceeded to restore GitLab from the backup taken earlier. The base image restored GitLab to 8.16.3 and the backup was from 8.16.4 which poses an issue because the versions must match. Installation of GitLab 8.16.4 failed; therefore, a system wide upgrade was performed on the GitLab host which bumped the GitLab version to 8.16.6. GitLab was then rolled back to 8.16.4 and the backup successfully restored. GitLab was then rolled forward to 8.16.6, re-added to salt and smoke checked.

It was never determined how GitLab was running for 14 days on a non-existent delta image while maintaining data integrity. On February 2nd the delta image was removed with an “rm -rf” command after a failed block commit during manual maintenance. There is clearly a design feature of KVM and external snapshots that I do not fully understand.

GitLab was successfully restored after 30 minutes of downtime. No data was lost as a result of the incident.

Action Items

Backups – GitLab automated backup frequency has been increased from 1x a day to 3x.

Process – A GitLab maintenance process has been instated to assure a backup is always available.

Cold Spare – A GitLab cold spare, git02, was created, salted, shutdown and ready to boot.

Timeline

On February 19th, 2017 PST…

01:54 – Manual GitLab backup taken.
01:55 – Nagios downtime set.
01:56 – GitLab rebooted.
01:57 – GitLab not coming back online…
01:59 – Determined GitLab was running from non existent delta image.
02:00 – Edited XML to boot from base image and booted GitLab.
02:01 – GitLab online from base image dating 2017-02-05 at 22:26.
02:05 – GitLab taken back offline due to 2-week-old data.
02:06 – Investigated root cause in attempt to restore machine image.
02:17 – Determined machine image restore was not possible.
02:18 – GitLab Restore failed due to GitLab version mismatch.
02:19 – GitLab 8.16.4 install failed for unknown reasons.
02:20 – GitLab system upgraded and GitLab upgraded to 8.16.6.
02:22 – Rolled GitLab installation back to version 8.16.4.
02:26 – Restored GitLab from backup on version 8.16.4.
02:28 – Upgraded GitLab to latest version 8.16.6 and verified integrity.
02:35 – Re-salted git01 since it was restored from a non-salted image.