Unleash the Power of EBSight for Optimal AWS Storage Management 🚀

Hey there, tech aficionados! 👋

Recently, AWS dropped a neat minor update – they started showing the full size of your EBS snapshots. Game changer! This isn’t just the incremental stuff; it’s the data footprint.

This is awesome because now we can really see what’s going on under the hood, and that’s where EBSight comes in. I whipped this up 🐍 to take advantage of this new snapshot information and give you some seriously insightful data about your volumes.

https://github.com/sjramblings/ebsight

EBSight is a tool that analyses your EBS volumes using the fresh FullSnapshotSizeInBytes field from AWS. Its goal is to provide you with the lowdown on your storage usage, performance, and where you can trim some fat.

We’re talking about tracking data growth, spotting inefficiencies, and getting recommendations for things like fsx for NetApp ONTAP.

Intrigued? Let’s dive deeper! 👇

Peeling Back the Layers: Understanding EBS Snapshots (The Right Way!) 🧅

EBS snapshots can be a bit mysterious. We know they’re important for backups and disaster recovery, but understanding how they work under the hood can feel like deciphering ancient hieroglyphics.

The key thing to remember is that EBS snapshots are incremental. This isn’t just tech jargon; it’s pretty clever and saves a lot of space (and money!).

Think of it this way: Imagine you have a 100GB EBS volume holding, say, 50GB of actual data. When you take your first snapshot, AWS doesn’t copy 100 GB; it only snapshots the 50GB of data you’re using.

And here’s the kicker: subsequent snapshots are even more efficient. They only store the changes you’ve made since the last snapshot. If you only change a bit of data, the next snapshot will be small. Unchanged blocks? They just get referenced from previous snapshots. It’s like magic, but it’s just smart engineering. ✨

Here’s where the new AWS feature and EBSight come into play. Before, figuring out the full size of your snapshots—the total data they represent—was kind of murky. You could see the incremental size but not the whole picture. But with this new FullSnapshotSizeInBytes field, AWS is giving us the full monty.

Even though snapshots are incremental and stored efficiently, this “full snapshot size” tells you the total amount of data present on your volume at the moment you took a snapshot. So, in our 100GB volume with 50GB data example, the “full snapshot size” will show 50GB, whether it’s your first or tenth snapshot.

EBSight uses this information to track data changes over time, giving you a crystal-clear view of your volume usage. No more guesswork! We’re talking real, actionable insights.

This is how you optimise your cloud spending and become a true cloud ninja. 🥷

Why You Need This in Your Life (and Your AWS Account) 🦸

So, we get that EBS snapshots are incremental, and now we can see their full size. Cool, right? But why do you need EBSight? What’s the real-world value here?

EBSight is your data detective, giving you superpowers to understand your EBS environment. Here’s how it helps:

1. Track Actual Data Usage vs. Allocated Size: EBSight shows you how much actual data you’re storing in your snapshots compared to the allocated size of your volumes. EBSight highlights these discrepancies so you can consider resizing those volumes and reclaiming those wasted resources. 💰
2. Monitor Data Change Patterns Over Time: Are your AWS Backups growing like crazy? Or are they relatively stable? EBSight tracks data change rates day by day based on the number of snapshots. This is gold for capacity planning. If you see a volume consistently growing by 10% daily, you need to keep an eye on it and potentially scale up (or optimise your application!). Conversely, if a volume is barely changing, you can be more aggressive with storage optimisation.
3. Calculate Storage Efficiency Metrics: EBSight crunches the numbers and gives you clear metrics like “Usage %” and “Daily Change Rate %”. These aren’t just random stats; they’re actionable insights. A low “Usage %” screams “resize me!”. A high “Daily Change Rate %” might indicate a need for more frequent snapshots or a closer look at your application’s data patterns.
4. Provide Cost Optimisation Recommendations: EBSight calculates estimated daily, monthly, and annual backup costs based on your snapshot sizes and change rates. Seeing these numbers in black and white can be a real eye-opener. It helps justify storage optimisation efforts and directly correlates them to cost savings. Plus, with the fsx for NetApp ONTAP recommendations (more on that later!), EBSight even helps you explore alternative storage solutions that might be more cost-effective in the long run.

In short, EBSight isn’t just another monitoring tool. It empowers you with the data you need to make smart decisions about your storage, save money, and keep your AWS environment running smoothly.

Under the Hood

Let’s get down to the nitty-gritty and explore the key features that make this tool tick. It’s not just about spitting out raw numbers; it’s about presenting information in a useful way, turning raw data into actionable intelligence.

Volume Analysis ⚙️

• Tracks Total Snapshot Sizes: Leveraging the shiny new FullSnapshotSizeInBytes field, EBSight grabs the full size of each snapshot for your volumes. It analyses up to the last 50 snapshots, which is the foundation for all the other calculations.
• Calculates Daily Change Rates: This is where things get interesting. EBSight compares snapshot sizes day by day to calculate the daily change rate for each volume. It shows both the absolute change in GiB and the percentage change. This is crucial for understanding how quickly your data is growing. A consistently high daily change rate might indicate a need for more efficient data management strategies or a potential bottleneck in your application.
• Monitors Volume Usage Percentages: EBSight calculates the “Usage %” by comparing the total snapshot size to the provisioned volume size. This gives you a clear picture of how efficiently you utilise your allocated storage. Volumes with low usage percentages are prime candidates for resizing.
• Detailed IOPS and Throughput Metrics: Storage size is not the only factor; performance matters, too! EBSight pulls performance metrics from CloudWatch, focusing on IOPS (Input/Output Operations Per Second) and Throughput (MBps). It breaks these down into read and write operations, giving you a granular view of your volume’s performance characteristics.

Performance Metrics

Storage size is essential, but performance is equally critical, especially for databases and applications that demand low latency and high throughput. EBSight goes beyond just size analysis and provides insights into your volume’s performance:

• P99 and Peak IOPS for Read/Write Operations: EBSight retrieves and displays both P99 (99th percentile) and Peak IOPS for read and write operations over the last 7 days. P99 IOPS indicates your typical high-end performance, while Peak IOPS shows you the maximum burst performance your volume has experienced. This is vital for identifying potential performance bottlenecks and ensuring your volumes are sized appropriately for your workload demands.
• Throughput Measurements in MBps: Similar to IOPS, EBSight provides P99 and Peak Throughput metrics in MBps for read and write operations. Throughput is crucial for applications that transfer large amounts of data, like backups or data warehousing.
• Queue Length Monitoring: Queue length is another key performance indicator. A consistently high queue length suggests that your volume might struggle to meet the demand, potentially degrading performance. EBSight monitors and displays queue lengths, helping you identify potential bottlenecks.
• Historical Performance Data (7-Day Period): All performance metrics are analysed over 7 days to give you a representative view of your volume’s typical performance. This historical perspective helps you identify trends and patterns that might not be apparent from real-time metrics.

Visualisation Options

Data is great, but visualising it makes it even better! EBSight offers several visualisation options to help you quickly grasp the key insights:

• ASCII-Based Bar Graphs for Size Comparisons: EBSight generates simple ASCII-based bar graphs right in your terminal for a quick visual comparison of volume vs. snapshot size. It’s a super-fast way to spot those over-provisioned volumes at a glance. Sometimes, a simple visual is all you need!
• Consolidated Summary Tables: EBSight presents all the key metrics—volume size, snapshot size, usage percentage, change rates, and performance metrics—in well-formatted, easy-to-read summary tables. These tables provide a consolidated view of all your volume analysis results.
• Optional CSV Export Functionality (-csv): Need to do more in-depth analysis or integrate EBSight’s data with other tools? No problem! EBSight supports exporting results to CSV format with the -csv option. This allows you to easily import the data into spreadsheets, databases, or other analysis platforms.

fsx for NetApp ONTAP Sizing Recommendations: Thinking Beyond EBS 💡

Sometimes, EBS might not be the optimal storage solution for all workloads. If you’re considering migrating to fsx for NetApp ONTAP (a super-powerful, feature-rich file storage service from AWS), EBSight can help!

• fsx for NetApp ONTAP Sizing Recommendations: With the -fsx option, EBSight takes your volume analysis data and provides sizing recommendations for fsx for NetApp ONTAP. It estimates the total storage required, recommended SSD size, sustained/peak IOPS and throughput requirements based on your EBS volume usage and performance patterns. This is incredibly helpful for migration planning and cost estimation if you’re exploring fsx as an alternative.

EBSight has features that give you a 360-degree view of your EBS volumes – size, performance, efficiency, and migration options. 🤓

Getting Your Hands Dirty

Okay, enough talk; let’s get practical! You should try out EBSight and see what it can do for your AWS environment. The good news is that it’s super easy to get up and running.

Basic Command

The simplest way to run EBSight is with the basic command:

python ebsight.py

Just fire this up in your terminal, and EBSight will guide you through the process. It will prompt you to enter the EC2 Instance ID you want to analyse. Once you provide that, EBSight gets to work, analysing all the EBS volumes attached to that instance.

Unleashing the Options

Want to dive deeper or customise the output? EBSight comes with a bunch of handy options you can use from the command line:

python ebsight.py [OPTIONS]

Here’s a breakdown of the available options and what they do:

• -verbose, v: Show Detailed Snapshot Information. Use this option to see the raw snapshot data, including snapshot IDs, start times, and sizes. Good for debugging or just geeking out on the details. 🤓
• -csv, c: Export Results to CSV. As mentioned, this option exports all the analysis results to a CSV file. Perfect for importing into spreadsheets or other data analysis tools. The CSV file will be created in the same directory where you run EBSight.
• -graph, g: Show ASCII Graphs. Enable this option to display those cool ASCII bar graphs that compare volume and snapshot sizes in your terminal output. Visuals, baby! 📊
• -profile, p: AWS Profile Name (default: ‘default’). If you’re using named profiles in your AWS CLI configuration (and you should be!), you can specify the profile with this option. For example, -profile my-dev-account. If you don’t specify a profile, EBSight will use the default profile.
• -fsx, f: Show fsx for NetApp ONTAP Recommendations. Want to see those fsx sizing recommendations? Just add this option! EBSight will include the fsx recommendations in its output.

Example Time! Let’s say you want to analyse an EC2 instance with ID i-0a1b2c3d4e5f6g7h8 using your AWS profile named demo-profile, and you want to see the ASCII graphs and fsx recommendations. Your command would look like this -python ebsight.py -p demo-profile -g -f

EBSight will then prompt you to enter the EC2 Instance ID (even though you could technically pass it as an argument, prompting makes it a bit more user-friendly for quick use). You’d enter i-0a1b2c3d4e5f6g7h8, and EBSight will do its magic! ✨

$ python ebsight.py -p demo-profile -g -f
Enter EC2 Instance ID: i-0a1b2c3d4e5f6g7h8

🖥️  Analyzing EC2 Instance: i-0a1b2c3d4e5f6g7h8
Name: demo-database-01
================================================================================

💾 Found 7 attached volumes

📀 Analyzing Volume: vol-0a1b2c3d4e5f6g7h1 (/dev/sda1)
================================================================================
🔍 Found 7 snapshots

   Volume Summary:
   ------------------------------------------------------------
   Metric                    Value                         
   ------------------------------------------------------------
   Volume Size               150.0 GiB                     
   Snapshot Total Size       79.8 GiB                      
   Usage %                   53.2%                         
   Snapshot Count            7                             
   Daily Change Rate         13.31 GiB (8.9%)              
   Total Changed Data        79.8 GiB                      
   Daily Backup Cost         $0.13
   Monthly Backup Cost       $3.99
   Annual Backup Cost        $47.90
   ------------------------------------------------------------

   📊 Size Comparison for vol-0a1b2c3d4e5f6g7h1
   ==============================================================
   Volume Size    ██████████████████████████████████████████████████ 150.00 GiB
   Snapshot Size  ██████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 79.83 GiB

📀 Analyzing Volume: vol-0a1b2c3d4e5f6g7h2 (/dev/sdb)
================================================================================
🔍 Found 7 snapshots

   Volume Summary:
   ------------------------------------------------------------
   Metric                    Value                         
   ------------------------------------------------------------
   Volume Size               210.0 GiB                     
   Snapshot Total Size       106.8 GiB                     
   Usage %                   50.8%                         
   Snapshot Count            7                             
   Daily Change Rate         17.79 GiB (8.5%)              
   Total Changed Data        106.8 GiB                     
   Daily Backup Cost         $0.18
   Monthly Backup Cost       $5.34
   Annual Backup Cost        $64.05
   ------------------------------------------------------------

   📊 Size Comparison for vol-0a1b2c3d4e5f6g7h2
   ==============================================================
   Volume Size    ██████████████████████████████████████████████████ 210.00 GiB
   Snapshot Size  █████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 106.76 GiB

📀 Analyzing Volume: vol-0a1b2c3d4e5f6g7h3 (/dev/sdc)
================================================================================
🔍 Found 7 snapshots

   Volume Summary:
   ------------------------------------------------------------
   Metric                    Value                         
   ------------------------------------------------------------
   Volume Size               2500.0 GiB                    
   Snapshot Total Size       1965.5 GiB                    
   Usage %                   78.6%                         
   Snapshot Count            7                             
   Daily Change Rate         327.58 GiB (13.1%)            
   Total Changed Data        1965.5 GiB                    
   Daily Backup Cost         $3.28
   Monthly Backup Cost       $98.28
   Annual Backup Cost        $1179.30
   ------------------------------------------------------------

   📊 Size Comparison for vol-0a1b2c3d4e5f6g7h3
   ==============================================================
   Volume Size    ██████████████████████████████████████████████████ 2500.00 GiB
   Snapshot Size  ███████████████████████████████████████░░░░░░░░░░░ 1965.51 GiB

📀 Analyzing Volume: vol-0a1b2c3d4e5f6g7h4 (/dev/sdd)
================================================================================
🔍 Found 7 snapshots

   Volume Summary:
   ------------------------------------------------------------
   Metric                    Value                         
   ------------------------------------------------------------
   Volume Size               201.0 GiB                     
   Snapshot Total Size       27.5 GiB                      
   Usage %                   13.7%                         
   Snapshot Count            7                             
   Daily Change Rate         4.58 GiB (2.3%)               
   Total Changed Data        27.5 GiB                      
   Daily Backup Cost         $0.05
   Monthly Backup Cost       $1.37
   Annual Backup Cost        $16.50
   ------------------------------------------------------------

   📊 Size Comparison for vol-0a1b2c3d4e5f6g7h4
   ==============================================================
   Volume Size    ██████████████████████████████████████████████████ 201.00 GiB
   Snapshot Size  ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 27.50 GiB

📀 Analyzing Volume: vol-0a1b2c3d4e5f6g7h5 (/dev/sde)
================================================================================
🔍 Found 7 snapshots

   Volume Summary:
   ------------------------------------------------------------
   Metric                    Value                         
   ------------------------------------------------------------
   Volume Size               50.0 GiB                      
   Snapshot Total Size       19.3 GiB                      
   Usage %                   38.6%                         
   Snapshot Count            7                             
   Daily Change Rate         3.22 GiB (6.4%)               
   Total Changed Data        19.3 GiB                      
   Daily Backup Cost         $0.03
   Monthly Backup Cost       $0.97
   Annual Backup Cost        $11.59
   ------------------------------------------------------------

   📊 Size Comparison for vol-0a1b2c3d4e5f6g7h5
   ==============================================================
   Volume Size    ██████████████████████████████████████████████████ 50.00 GiB
   Snapshot Size  ███████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 19.31 GiB

📀 Analyzing Volume: vol-0a1b2c3d4e5f6g7h6 (/dev/sdf)
================================================================================
🔍 Found 7 snapshots

   Volume Summary:
   ------------------------------------------------------------
   Metric                    Value                         
   ------------------------------------------------------------
   Volume Size               203.0 GiB                     
   Snapshot Total Size       53.0 GiB                      
   Usage %                   26.1%                         
   Snapshot Count            7                             
   Daily Change Rate         8.83 GiB (4.4%)               
   Total Changed Data        53.0 GiB                      
   Daily Backup Cost         $0.09
   Monthly Backup Cost       $2.65
   Annual Backup Cost        $31.80
   ------------------------------------------------------------

   📊 Size Comparison for vol-0a1b2c3d4e5f6g7h6
   ==============================================================
   Volume Size    ██████████████████████████████████████████████████ 203.00 GiB
   Snapshot Size  █████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 53.00 GiB

📀 Analyzing Volume: vol-0a1b2c3d4e5f6g7h7 (/dev/sdg)
================================================================================
🔍 Found 7 snapshots

   Volume Summary:
   ------------------------------------------------------------
   Metric                    Value                         
   ------------------------------------------------------------
   Volume Size               52.0 GiB                      
   Snapshot Total Size       46.3 GiB                      
   Usage %                   89.0%                         
   Snapshot Count            7                             
   Daily Change Rate         7.72 GiB (14.8%)              
   Total Changed Data        46.3 GiB                      
   Daily Backup Cost         $0.08
   Monthly Backup Cost       $2.32
   Annual Backup Cost        $27.78
   ------------------------------------------------------------

   📊 Size Comparison for vol-0a1b2c3d4e5f6g7h7
   ==============================================================
   Volume Size    ██████████████████████████████████████████████████ 52.00 GiB
   Snapshot Size  ████████████████████████████████████████████░░░░░░ 46.30 GiB

   Volume Analysis Summary:
   ====================================================================================================================================================================================
   Volume ID                 Mount    Size     Used %   P99 IOPS (R/W)        Peak IOPS (R/W)       P99 MBps (R/W)        Peak MBps (R/W)       Queue   
   ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   0a1b2c3d4e5f6g7h1        sda1     150.0    53.2    35/21                 35/21                 1.1/3.0               1.1/3.0               0.59    
   0a1b2c3d4e5f6g7h2        sdb      210.0    50.8    2/1                   2/1                   0.1/0.1               0.1/0.1               0.02    
   0a1b2c3d4e5f6g7h3        sdc      2500.0   78.6    112/26                113/26                5.2/1.3               5.2/1.3               0.85    
   0a1b2c3d4e5f6g7h4        sdd      201.0    13.7    30/0                  30/0                  4.1/0.0               4.1/0.0               0.48    
   0a1b2c3d4e5f6g7h5        sde      50.0     38.6    22/27                 22/27                 1.4/1.3               1.4/1.3               0.36    
   0a1b2c3d4e5f6g7h6        sdf      203.0    26.1    3/18                  3/18                  0.2/0.5               0.2/0.5               0.18    
   0a1b2c3d4e5f6g7h7        sdg      52.0     89.0    0/1                   0/1                   0.0/0.0               0.0/0.0               0.01    
   ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   TOTALS:                          3366.0            204/94                206/94                12.0/6.1              12.1/6.1         
   ====================================================================================================================================================================================

   📊 fsx for NetApp ONTAP Sizing Recommendations:
   --------------------------------------------------------------------------------
   Total Storage Required (GB):    3366
   Recommended SSD Size (GB):      383  (11.4% of total)
   Sustained IOPS Required:        298
   Peak IOPS Required:             300
   Sustained Throughput (MBps):    18.1
   Peak Throughput (MBps):         18.2
   --------------------------------------------------------------------------------

✅ Analysis completed for all volumes.

Wrapping Up: Go Forth and Optimise! 🚀

EBSight is all about bringing clarity and control to your EBS storage. It leverages AWS’s awesome new FullSnapshotSizeInBytes feature to give you real, actionable insights into your volume usage, performance, and potential cost savings.

From tracking data growth and identifying over-provisioned volumes to providing fsx migration recommendations, EBSight is designed to be your go-to companion for EBS optimisation. It’s easy to install, simple to use, and packed with features that can make a real difference in your AWS bill and operational efficiency.

So, what are you waiting for? Clone the repo, set up your environment, and start exploring your EBS volumes with EBSight! I genuinely hope this tool helps someone else manage their cloud storage.

Let me know in the comments below how you’re using EBSight and if you have any feedback—I’m always looking for ways to improve it! You can raise an issue or PR in the public repo here.

Hope this helps someone else.