Pages: [1]   Go Down

Author Topic: New PC - suggestions for storage set-up?  (Read 4955 times)

AndrewKulin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 342
    • http://www.andrewkulin.com
New PC - suggestions for storage set-up?
« on: October 12, 2012, 08:16:22 am »

I am in process of building a new multi-use PC (CS6/LR4 etc., some gaming, and general use (MS Office, Web, etc.).   General characteristics of the new machine are X79 M/B, i7 3930K CPU, 64 GB RAM, Win 7 x64 Ult.  Some components from my existing set up will be transferred over into the new machine (ATI HD6970 GPU, 120 GB SSD (Corsair Force 3 GT - currently used as scratch), and external HDD enclosure, with two x 2 GB WD Caviar Black HDDs set up as Raid 1 for all photo storage, connected to existing via USB 3, + 1 or 2 other HDDs used for things like document storage, and the two monitors).  I am looking to optimize the machine for photo tasks over the other two (I am sure that whatever resulting gaming characteristics are will surpass my current rig)

The C-drive on my existing PC with Win 7x64 Ult. currently stands at 350 GB storage used, even though I have set up Windows with photos (incl. my photos) saved on the RAID enclosure, and My Documents stored on a separate HDD (as are whatever limited videos I may have).  So the C-drive essentially holds the O/S and program files yet is at 350 GB storage so far.  And with that in mind, I purchased a 512 GB SSD (Ocz Vertex 4) for use as my C-drive on the new rig, yet I keep reading on the web recommendations for O/S drive to be small sized (e.g., 64 GB, 120 GB).

My original thinking on this was:

C: - 512 GB SSD (boot) with O/S (and software)
E: - 2 TB photo drive (external Raid 1 enclosure) connected via USB 3 (e-sata also possible)
F: - 1 TB HDD for document storage
Z: - 120 GB SSD as Scratch disk (noting that I would also investigate using a portion of the RAM as a RAMDISK as well (for Scratch) to see if that is useful or not)

Or would this be a better approach:

C: - 120 GB SSD (boot) with O/S only
D: - 512 GB SSD for program files (I would need to look up how that can be done automatically in Win 7 - i.e. have it recognize that the two program file directories are not located on C: if that is possible)
E: - 2 TB photo drive (external Raid 1 enclosure) connected via USB 3 (e-sata also possible)
F: - 1 TB HDD for document storage
Z: - partition a portion of the 512 GB SSD as Scratch disk (this may be nonsense?)
         
Thoughts/advice would be appreciated




Logged
[size=12p

Rhossydd

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3369
    • http://www.paulholman.com
Re: New PC - suggestions for storage set-up?
« Reply #1 on: October 12, 2012, 10:08:20 am »

..... So the C-drive essentially holds the O/S and program files yet is at 350 GB storage so far. 
350gb seems an awful lot. Are you sure you really need all that on your system drive ?
My system is pretty much fully loaded with lots of imaging and video software and that only adds up to 120gb.
Are you loading lots of games software on the system drive drive ? Games and leisure stuff probably doesn't benefit overly from being on an SSD, so just put it on a normal drive instead.
Quote
yet I keep reading on the web recommendations for O/S drive to be small sized (e.g., 64 GB, 120 GB).
I've never come across that. Maybe it's a cost saving recommendation ?
The one aspect I have heard of with respect to SSD sizing is try to keep them no more 65% full.
Logged

AndrewKulin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 342
    • http://www.andrewkulin.com
Re: New PC - suggestions for storage set-up?
« Reply #2 on: October 13, 2012, 12:09:45 pm »

Your reply made me think about this and take a critical look at what is being saved on my OS drive.  I found this little gem of a freeware program (TreeSize Free) that scans your drive and lists out directories in order of space usage (incl. subdirectories) which is much more useful than windows explorer (see attached).  So what I have found is:

  • Lightroom catalog previews (47 GB) and backups (33 GB) are being saved on C in my pictures (so I'll set up LR to save all that on a separate drive (oops))
  • The 52 GB of Appdata is mostly Adobe Camera Raw Cache (43 GB) and I'll need to look into how I can reduce this automatically and also get it saved it on some other drive (if possible)
  • Temp Files make up about about 69 GB of the 99 GB Windows (so again research into what I can do to reduce these files sizes)
  • Pagefile is 8 GB - current system has 8 GB RAM - this I wonder if I can reduce or not, particularly as new system will have 64 GB RAM (anyone have any advice about this? because 64GB pagefile would take up a large part of the OS drive otherwise)
  • Hibernation file is 6GB and I think I can just turn that off if needed
  • 51 GB of program files = Games which  I can either delete or be more diligent in directing install directories onto another drive (though some installers don't allow the option)

I think what I'd like to do now is use the 120 GB Corsair Force 3 drive for the OS (but if pagefile has to be 64 GB then probably not) and partition the 512 GB SSD into two (or more) drives - one as scratch and one or more to store things like LR catalogs and Camera Raw cache, games and other program files.  But would using the same SSD for scratch and other purposes not be recommended? (particularly the photo editing applications, which would also be using the scratch drive)

Logged
[size=12p

Rhossydd

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3369
    • http://www.paulholman.com
Re: New PC - suggestions for storage set-up?
« Reply #3 on: October 13, 2012, 01:01:11 pm »

Moving the LR catalogue and previews to a separate SSD drive and deleting unnecessary backups will free up a load of space.
Yes, you can move the ACR cache onto the LR SSD, If space isn't an issue keep a big ACR cache has benefits I think.

Quote
Temp Files make up about about 69 GB of the 99 GB Windows (so again research into what I can do to reduce these files sizes)
That's a HUGE waste of space, get in there and free up some space. My system has 220mb in the Temp folder.

I've never tweaked the pagefile size, but you ought to be able to limit that.

Logged

John.Murray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 886
    • Images by Murray
Re: New PC - suggestions for storage set-up?
« Reply #4 on: October 13, 2012, 01:48:44 pm »

Andrew:  I would recommend the 2nd approach using a high quality SSD for the System Drive such as an Intel 510 Series or even a Revodrive.

Since you are contemplating a fresh install, you may want to consider moving User Profiles off the System Drive using sysprep:

1) Have both the System Drive and the Drive you want User Profiles on (the 512GB SSD would be perfect) installed before you begin.  Make sure your boot order is set to use the System Drive.

2) During your Windows installation, at the point you are prompted to create a username/password STOP!  Press SHIFT+CTRL+F3 which will reboot the system into Audit Mode.  Press Cancel at the Audit Dialog

3) From an Administrative Command Prompt (start | run | cmd [ctrl]+[shft]+[enter]) run:

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>cd sysprep
C:\Windows\system32\sysprep>sysprep /audit /reboot /unattend:f:\profiles.xml

f: would be the drive letter of a thumbdrive containing the following text making up profiles.xml:
<?xml version="1.0" encoding="utf-8"?>
   <unattend xmlns="urn:schemas-microsoft-com:unattend">
      <settings pass="oobeSystem">
         <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <FolderLocations>
               <ProfilesDirectory>d:\Users</ProfilesDirectory>
               <ProgramData>d:\ProgramData</ProgramData>
            </FolderLocations>
         </component>
      </settings>
   <cpi:offlineImage cpi:source="wim:E:/sources/install.wim#Windows 7 PROFESSIONAL" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>


Note that each line begins with a < and ends with a >  in other words, there should be 12 lines.  I'll assume you are installing a 64Bit O/S in Line 4, but note the Version in Line 11 ....

4) After running Sysprep, Windows will reboot back into Audit Mode, press OK to reboot back into normal operation
5) All user profiles wil now be located on the D: drive, including User Preferences and Application Data temp files......

If you are interested in moving existing user profiles (not just My Docs, My Pictures, etc) to another volume you can set a NTFS Directory Junction Point pointing to the new location:

http://www.starkeith.net/coredump/2009/05/18/how-to-move-your-windows-user-profile-to-another-drive/
« Last Edit: October 19, 2012, 02:09:32 pm by John.Murray »
Logged

AndrewKulin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 342
    • http://www.andrewkulin.com
Re: New PC - suggestions for storage set-up?
« Reply #5 on: October 13, 2012, 02:27:21 pm »

Andrew:  I would recommend the 2nd approach using a high quality SSD for the System Drive such as an Intel 510 Series or even a Revodrive.

Any recommended size?  120 GB?

Since you are contemplating a fresh install, you may want to consider moving User Profiles off the System Drive using sysprep:

1) Have both the System Drive and the Drive you want User Profiles on (the 512GB SSD would be perfect) installed before you begin.  Make sure your boot order is set to use the System Drive.

2) During your Windows installation, at the point you are prompted to create a username/password STOP!  Press SHIFT+CTRL+F3 which will reboot the system into Audit Mode.  Press Cancel at the Audit Dialog

3) From a Command Prompt run:

sysprep audit /reboot /unattend:f:\profiles.xml

f: would be the drive letter of a thumbdrive containing the following text making up profiles.xml:
<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
<settings pass="oobeSystem">
<component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<FolderLocations> <ProfilesDirectory>d:\Users</ProfilesDirectory> <ProgramData>d:\ProgramData</ProgramData>
</FolderLocations> </component>
</settings>
<cpi:offlineImage cpi:source="wim:E:/sources/install.wim#Windows 7 PROFESSIONAL" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>

Note that each line begins with a < and ends with a >  in other words, there should be 12 lines.  I'll assume you are installing a 64Bit O/S in Line 4, but note the Version in Line 11 ....

4)  After running Sysprep, Windows will reboot back into Audit Mode, press OK to reboot back into normal operation
5) All user profiles wil now be located on the D: drive, including User Preferences and Application Data temp files......

A few questions regarding the information above:

1.  The code in the grey box comes out as 9 lines and there are more than 12 pairings of the < & > symbols so unclear exactly how to convert into 12 lines should I go this route
2.  I would be running Windows Ultimate 64 so on line 11 I should replace "PROFESSIONAL" with "ULTIMATE" ?
3.  Is it necessary that this text file on thumb drive be the only thing on the drive?  And I assume the text file name is "profiles.xml".
4.  During windows installation process will system installation have progressed enough to even recognize a thumb-drive sitting in a USB port?  I assume so as sounds to me like you have done this often enough, and so will there be a prompt on the screen that identifies to me what drive letters are available and what type of drive it is so that I can correctly identify the drive and file?  Presumably it could be something entirely different from "f".  And do I have this thumb drive installed in the USB port right from the beginning of the installation process?

Thanks
Logged
[size=12p

John.Murray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 886
    • Images by Murray
Re: New PC - suggestions for storage set-up?
« Reply #6 on: October 13, 2012, 02:37:43 pm »

Andrew

1) I just touched the contents of profile.xml, should be displaying correctly as 12 lines
2) Yes - replace PROFESSIONAL with ULTIMATE
3) profiles.xml can be located anywhere, as long as you can get to it from the Windows Audit Mode command prompt.  It does not need to be the only file at that location
4) Yes, as long as the thumbdrive is inserted at the beginning of the windows install, I personally do all my Windows Installs from a thumbdrive -here's an article on how to make one:

http://imagesbymurray.com/technology/47-install-windows-from-usb

And yes, you'll need to determine what "drive" letter the thumb drive occupies - easily sorted out in the Audit environment....
Logged

tived

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 714
    • http://
Re: New PC - suggestions for storage set-up?
« Reply #7 on: October 17, 2012, 08:45:56 pm »

Thanks John, thats a great little trick - wished I had known this earlier - i find it so frustrating
thanks very much

Henrik

Andrew:  I would recommend the 2nd approach using a high quality SSD for the System Drive such as an Intel 510 Series or even a Revodrive.

Since you are contemplating a fresh install, you may want to consider moving User Profiles off the System Drive using sysprep:

1) Have both the System Drive and the Drive you want User Profiles on (the 512GB SSD would be perfect) installed before you begin.  Make sure your boot order is set to use the System Drive.

2) During your Windows installation, at the point you are prompted to create a username/password STOP!  Press SHIFT+CTRL+F3 which will reboot the system into Audit Mode.  Press Cancel at the Audit Dialog

3) From a Command Prompt run:

sysprep audit /reboot /unattend:f:\profiles.xml

f: would be the drive letter of a thumbdrive containing the following text making up profiles.xml:
<?xml version="1.0" encoding="utf-8"?>
   <unattend xmlns="urn:schemas-microsoft-com:unattend">
      <settings pass="oobeSystem">
         <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <FolderLocations>
               <ProfilesDirectory>d:\Users</ProfilesDirectory>
               <ProgramData>d:\ProgramData</ProgramData>
            </FolderLocations>
         </component>
      </settings>
   <cpi:offlineImage cpi:source="wim:E:/sources/install.wim#Windows 7 PROFESSIONAL" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>


Note that each line begins with a < and ends with a >  in other words, there should be 12 lines.  I'll assume you are installing a 64Bit O/S in Line 4, but note the Version in Line 11 ....

4) After running Sysprep, Windows will reboot back into Audit Mode, press OK to reboot back into normal operation
5) All user profiles wil now be located on the D: drive, including User Preferences and Application Data temp files......

If you are interested in moving existing user profiles (not just My Docs, My Pictures, etc) to another volume you can set a NTFS Directory Junction Point pointing to the new location:

http://www.starkeith.net/coredump/2009/05/18/how-to-move-your-windows-user-profile-to-another-drive/
Logged

jonathanlung

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 70
Re: New PC - suggestions for storage set-up?
« Reply #8 on: October 18, 2012, 03:02:38 pm »

I agree that 120GB for a sys drive + programs is overkill. But then again, I'm writing this on a MacBook Air where everything, including my data, lives in 128GB!

I've been out of touch with the Windows world for almost a decade but you should, unless Windows has regressed, be able to mount a physical drive as a directory on a computer. That is, you could have individual programs all live in C:\Program Files\ -- but individual programs would reside on different physical disks.

If you're getting two large SSD drives, it might make sense to partition one (or both) of them to create a RAID 0 (striped) disk in software for your scratch disk. Then again, I'm not sure if the software overhead will outweigh the performance increase. So just an idea that someone else might look at tackling.
Logged

AndrewKulin

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 342
    • http://www.andrewkulin.com
Re: New PC - suggestions for storage set-up?
« Reply #9 on: October 18, 2012, 09:22:41 pm »

For anone trying this, I found a small error in the sysrep instructions (#3).  It should have read as follows (bold text added):

3) From a Command Prompt run:

C:\Windows\System32\sysprep sysprep.exe /audit /reboot /unattend:f:\profiles.xml

You need to run the executable from the directory shown above.

It worked fine after I figured out the missing info.
Logged
[size=12p

Sareesh Sudhakaran

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 546
    • The Indie Farm
Re: New PC - suggestions for storage set-up?
« Reply #10 on: October 19, 2012, 05:07:30 am »

I am in process of building a new multi-use PC (CS6/LR4 etc., some gaming, and general use (MS Office, Web, etc.). 

...yet I keep reading on the web recommendations for O/S drive to be small sized (e.g., 64 GB, 120 GB).

My original thinking on this was:

C: - 512 GB SSD (boot) with O/S (and software)
E: - 2 TB photo drive (external Raid 1 enclosure) connected via USB 3 (e-sata also possible)
F: - 1 TB HDD for document storage
Z: - 120 GB SSD as Scratch disk (noting that I would also investigate using a portion of the RAM as a RAMDISK as well (for Scratch) to see if that is useful or not)

Or would this be a better approach:

C: - 120 GB SSD (boot) with O/S only
D: - 512 GB SSD for program files (I would need to look up how that can be done automatically in Win 7 - i.e. have it recognize that the two program file directories are not located on C: if that is possible)
E: - 2 TB photo drive (external Raid 1 enclosure) connected via USB 3 (e-sata also possible)
F: - 1 TB HDD for document storage
Z: - partition a portion of the 512 GB SSD as Scratch disk (this may be nonsense?)
         
Thoughts/advice would be appreciated

This is how I would do it:

C: 160 or 256 GB SSD (for OS, Adobe and Apps only - no data)

I like to keep 50% of my OS free. Ideally this should be no less than 30%.
Future file systems are going to need more resources than NTFS
You could partition this drive for dual boot or different file systems, etc
You could keep more than one 'image' of your system with a bigger drive

D: Game Drive - depending on how many games you want. This drive will also be used for temp game files so if you can afford SSDs I've heard they offer excellent speed advantage over 7,200rpm drives. I don't have much experience on gaming infrastructure.

E: 512 SSD for temp/page/cache drive. You can set this drive in Adobe's settings and it will use this drive exclusively for this purpose. All your project files will also reside here. When you're done, these files can be archived.

F: 1 TB drive for documents and work data

NAS: Take your other drives and setup a FreeNas on the ZFS File system in RAID Z2 (RAID 6 equivalent), or use Windows Home Server. If you want to bunch up multiple types of drives in RAID, and don't want to learn or get your hands dirty with new technology, get a Synology system. You can use parts from your older computer for the NAS, if most of them is in working condition. The older computer can also be a render farm - for batch processing in the background.

The NAS becomes your 'Read drive and backup drive' if you're working with images. If you're working with video, then I suggest you use a separate Read drive, in which case the RAID 1 setup is good enough, running on software RAID.

The NAS can also hold your movies, music, and can feed a HTPC if you want it to. A typical low-budget NAS should atleast have 4 bays (for future proofing and expansion) and support GigE. It should give a typical speed of 50 MB/s, and can be transmitted wireless.

Over and above this, use either Google Drive or Amazon S3 to backup your critical stuff. I strongly recommend more than one cloud storage service. I use GD, S3 and iCloud. I'm also seriously considering Backblaze.

In this way, your media files are separate and always ready for use by different systems (computer, ipad, mobile, internet, wi-fi, etc). Your game drive is isolated. In the future, you don't have to start from scratch if one part needs to change.

Hope this helps.
Logged
Get the Free Comprehensive Guide to Rigging ANY Camera - one guide to rig them all - DSLRs to the Arri Alexa.

John.Murray

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 886
    • Images by Murray
Re: New PC - suggestions for storage set-up?
« Reply #11 on: October 19, 2012, 02:05:15 pm »

Thanks Andrew for pointing that out - I corrected the post to reflect the correct information :)
Logged
Pages: [1]   Go Up