{"id":194,"date":"2019-01-14T12:54:52","date_gmt":"2019-01-14T17:54:52","guid":{"rendered":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/?page_id=194"},"modified":"2026-11-05T16:24:00","modified_gmt":"2026-11-05T21:24:00","slug":"storage-technology-options","status":"publish","type":"page","link":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/docs\/file-storage\/storage-technology-options\/","title":{"rendered":"Storage Technology Options"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"194\" class=\"elementor elementor-194\" data-elementor-post-type=\"page\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-6bf49de3 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"6bf49de3\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-27b75677\" data-id=\"27b75677\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1a5b2dd0 elementor-widget elementor-widget-text-editor\" data-id=\"1a5b2dd0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div id=\"content-head\" class=\"row-fluid content-head\"><h2>Storage Information on CS Linux Systems<\/h2><p id=\"page-title\" class=\"title\">This page describes various storage facilities provided by LCSR for your use. However, you can use the same technology to set up your file servers or to make files on your server or desktop available from other systems.<\/p><p>Note that LCSR is moving toward a few file servers, all accessible from all systems that participate in the LCSR Kerberos system. This includes generally available systems such as iLab systems and most research systems.<\/p><p><strong>Note<\/strong>: Our machines use automount, which means the file system is mounted as you access it. To access the non-mounted file system, just cd to it. For example, to access <code>\/common\/users\/<i>$USER,<\/i><\/code> You type: <code>cd \/common\/users\/<i>$USER<\/i><\/code><\/p><details open=\"open\"><summary><b>CS Storage Summary<\/b><\/summary><div id=\"content-top\" class=\"row-fluid content-top\"><div class=\"region region-content-top clearfix\"><div id=\"block-system-main\" class=\"clearfix block block-system\"><div class=\"content\"><article id=\"node-321\" class=\"node node-resource-page clearfix\"><div class=\"content\"><div class=\"field field-name-body field-type-text-with-summary field-label-hidden\"><div class=\"field-items\"><div class=\"field-item even\"><table style=\"border: 1px solid grey;padding: 15px;text-align: left\" data-darkreader-inline-border-top=\"\" data-darkreader-inline-border-right=\"\" data-darkreader-inline-border-bottom=\"\" data-darkreader-inline-border-left=\"\"><caption><strong>CS Storage Summary<\/strong><\/caption><thead><tr><th style=\"width: 250px\">Storage<\/th><th style=\"width: 100px\">Type<\/th><th style=\"width: 100px\">Backed up<\/th><th style=\"text-align: center;width: 250px\"><p>Disk Quota<\/p><hr \/><p>Student, Affiliate\/PhD, Faculty<\/p><\/th><\/tr><\/thead><tbody><tr><td colspan=\"4\"><hr \/><\/td><\/tr><tr><td><code>\/common\/home\/<\/code><\/td><td>SSD<\/td><td>Yes<\/td><td>50GB \/ 200GB<\/td><\/tr><tr><td><code>\/common\/users\/<\/code><\/td><td>SSD<\/td><td>Yes<\/td><td>250GB \/ 2TB<\/td><\/tr><tr><td><code>\/fac\/users\/<\/code><\/td><td>SSD<\/td><td>Yes<\/td><td>100GB (Faculty only)<\/td><\/tr><tr><td><code>\/freespace\/local<\/code><\/td><td>SSD<\/td><td>No<\/td><td>None<\/td><\/tr><tr><td><code>\/filer\/tmp1<\/code><\/td><td>SSD<\/td><td>No<\/td><td>None<\/td><\/tr><tr><td><code><em>\/research\/projects\/{projectName}<\/em><\/code><\/td><td>Non SSD<\/td><td>Yes<\/td><td>project-specific<\/td><\/tr><tr><td><code>\/research\/archive<\/code><\/td><td>Non SSD<\/td><td>Yes<\/td><td>None<\/td><\/tr><\/tbody><\/table><p>Most systems in computer science have access to the following:<\/p><ul><li>Home directories [<code>\/common\/home<\/code>] for general-access systems are hosted in an SSD (actually <strong>NVMe<\/strong>) based file server with <strong>40 Gbps<\/strong> networking. This is intended for projects that need high performance.<\/li><li><code>\/filer\/tmp1<\/code> &#8211; intended as a working space if you need more storage for a short period. <strong>SSD <\/strong>storage has no quota but is not backed up. It will be cleared if filled out, and files may be removed at the beginning of every semester and summer.<\/li><li>Local storage, usually on <code>\/freespace\/local<\/code>. This is <strong>SSD<\/strong> storage (i.e., reasonably fast), but it is not backed up and may be cleared out when it fills and between semesters. This is intended for short-term working storage.<\/li><li><code>\/research\/archive<\/code>. This is for data that funding agencies require to be kept. This will eventually be merged\u00a0 <code>\/common\/users<\/code> into a single system for files that don&#8217;t need SSD. Please get in touch with us if you need this service.<\/li><\/ul><p>Essentially, <code>\/common\/home<\/code>It is the primary storage for all users, including research users,\u00a0 <code>\/common\/users<\/code> for files not currently being used heavily. <code>\/common\/home<\/code> has fast disks. <code>\/common\/users<\/code> will have a larger capacity but with traditional disks.<\/p><p style=\"text-align: center\">\u00a0<\/p><\/div><\/div><\/div><\/div><\/article><\/div><\/div><\/div><\/div><\/details><details open=\"open\"><summary><b>Technology We use<\/b><\/summary><p>Our underlying approach is NFS, a network file system that Unix and Linux support, and to some extent, MacOS and Windows. Specifically, we use NFS version 4 with Kerberos authentication. Version 4 with Kerberos allows the incorporation of systems run by different system administrators, which may not always have consistent UID and GIDs and may have varying levels of security.<\/p><p>We also recommend setting up the \/net virtual file system. That allows you to access files on any system that supports it without having your system administrator add a mount. Thus, it creates the illusion that file systems across all our servers are a part of a single, combined namespace.<\/p><p>Systems run by LCSR already have the necessary software. For information on integrating your systems with our Kerberos infrastructure, see <a href=\"http:\/\/services.cs.rutgers.edu\/kerberize.html\" target=\"_blank\" rel=\"noopener\">&#8220;Setting up Kerberos and Related Services.<\/a>&#8220;<\/p><\/details><details open=\"open\"><summary><span style=\"color: #800000\" data-darkreader-inline-color=\"\"><b>Do you need more space?<\/b><\/span><\/summary><p style=\"text-align: left\">See the Summary section below to determine how much disk quota is available for you. To see how much space is used for each available filesystem,<\/p><ul><li>open a <em>terminal<\/em> window <strong>or<\/strong> connect via <em><a href=\"https:\/\/www.howtogeek.com\/311287\/how-to-connect-to-an-ssh-server-from-windows-macos-or-linux\/\" target=\"_blank\" rel=\"noopener\">SSH client<\/a><\/em> and<\/li><li>type: <code>quota -vs<\/code>, and you will see something similar to the below<\/li><\/ul><p><img decoding=\"async\" class=\"size-full wp-image-4657 aligncenter\" src=\"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-content\/uploads\/sites\/45\/2021\/06\/quota-1.png\" alt=\"\" width=\"500\" height=\"98\" srcset=\"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-content\/uploads\/sites\/45\/2021\/06\/quota-1.png 500w, https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-content\/uploads\/sites\/45\/2021\/06\/quota-1-300x59.png 300w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/p><p>If you run out of disk space in your <em>home directory <\/em>(User home directory is<code> \/common\/home<\/code>), your current usage will be close to your quota.<\/p><p>To find out where your disk space went, type: <code>du -a ~ | sort -rn | more<\/code>. This command will list the <i>total space<\/i> used for each directory you own, with the most usage listed at the top.<\/p><p>In most cases, the hidden .cache<span class=\"s1\">, <\/span>.local\/lib<span class=\"s1\">, <\/span>.local\/share\/Trash<span class=\"s1\">, <\/span>.conda<span class=\"s1\">, <\/span>.vscode-server<span class=\"s1\">, <\/span>~\/vmware, other .local directories, and more are growing out of bounds and need to be cleaned up, or <em>you forgot to empty your Trash<\/em>.\u00a0<\/p><p>If you run out of space in your home directory, please utilize <code>\/freespace\/local<\/code>, <code>\/common\/users\/<em>your_netid<\/em><\/code> storage described in sections B and C below.<\/p><hr \/><\/details><h3>Storage available with Computer Science<\/h3><details open=\"open\"><summary><b>A.Home Directories<\/b><\/summary><p>Users&#8217; home directories on our generally available systems are on the LCSR machines named <strong>communis.lcsr<\/strong>. For Quota info, see the Summary section above. Each machine in its cluster shares the same home directory storage and is accessible at:<\/p><ul><li><code>\/common\/home<\/code> for iLab and general research machines.<\/li><li><code>\/fac\/users<\/code> for Faculty machines.<\/li><\/ul><p>These directories are backed up, and lost files are recoverable. Essentially, we keep backups for up to two months and have a fallback position in case of hardware failure.)\u00a0<\/p><p>Accounts are given home directory quota limits to share this very expensive resource. If your initial quota is too small, consider other shared disks like <code>\/common\/users<\/code>, <code>\/filer\/tmp1,<\/code> and <code>\/freespace\/local<\/code> below.<\/p><p>If you are working with a faculty member on a project, make sure you store all your work in your research-specific storage and not in your home directory. This storage should be accessible from any iLab machines. Please ask the faculty you work with for access. If the research has no\u00a0<span style=\"margin: 0px;padding: 0px\">storage, the faculty member can ask the <a href=\"https:\/\/resources.cs.rutgers.edu\/support\/\" target=\"_blank\" rel=\"noopener\"><em><strong>CS HelpDesk<\/strong><\/em><\/a> for\u00a0<\/span>storage.<\/p><\/details><details open=\"open\"><summary><b>B. Shared Local Filesystems<\/b><\/summary><ul><li><code>\/freespace\/local<\/code> (available only on iLab\/Grad\/Fac desktop machines)<br \/>Most machines have extra non-quota disk space in a partition called<code>\/freespace\/local<\/code>. You should note the filesystem restrictions below.<br \/>&#8211; Files are <strong>not backed up<\/strong><br \/>&#8211; This storage has no quota but is limited to the available space.<br \/>&#8211; Files in <code>\/freespace\/local<\/code> are automatically deleted when machines are reinstalled without prior warning.<br \/>&#8211; All files may be removed between the end and start of a semester or <em>when we run out of space <\/em>without prior warning.<\/li><\/ul><\/details><details open=\"open\"><summary><b>C. Computer Science Remote Filesystems<\/b><\/summary><ul><li><code>\/common\/home<\/code><br \/>&#8211; This is the default home directory for iLab and general CBIM machines.<br \/>&#8211; Based on a Linux server using the ZFS file system and NVMe storage.<br \/>&#8211; It is backed up daily. Daily snapshots are kept for 59 days. These can be used to recover files that were accidentally deleted.<br \/>&#8211; Researchers may request special project directories.<\/li><li><code>\/common\/users<\/code><br \/>&#8211; Can be accessed in \/common\/users\/<em>your_netid or <code>\/common\/users\/<i>$USER<\/i><\/code><\/em><br \/>&#8211; It is accessible from all systems participating in LCSR&#8217;s Kerberos, i.e., general-access and most research systems.<br \/>&#8211; This file system has a <strong>single daily backup<\/strong> and no snapshots.<\/li><li><code>\/fac\/users<\/code><br \/>&#8211; Can be accessed in \/fac\/users\/<em>your_netid or <code>\/fac\/users\/<i>$USER<\/i><\/code><\/em><br \/>&#8211; It is accessible only from faculty systems participating in LCSR&#8217;s Kerberos.<\/li><li><code>\/filer\/tmp1<\/code><br \/>&#8211; accessible from all generally available systems. This uses SSD storage.<br \/>&#8211; There is no quota, but it is limited to available space.<br \/>&#8211; is <strong>not backed up<\/strong>.<br \/>&#8211; is <strong>not<\/strong> intended to be permanent storage.<br \/>&#8211; <span style=\"color: #800000\">is cleared when filled<\/span> and may be cleared at the beginning of every semester and summer.<\/li><li>If you need a restore from backed-up storage, please ask the <a href=\"https:\/\/resources.cs.rutgers.edu\/support\/\" target=\"_blank\" rel=\"noopener\"><em><strong>CS HelpDesk<\/strong><\/em><\/a> and specify the full path of the item to be restored from a specific point in time.<\/li><\/ul><\/details><details open=\"open\"><summary><b>D. Special Storage For Projects<\/b><\/summary><ul><li><strong>Project space on our filesystems<\/strong><br \/>A larger space can be arranged for individuals or groups needing aggressively backed-up space. This space is generally subject to the same policy as our home directories. (Essentially, we keep backups for up to two months and have a fallback position in case of hardware failure.) This <em>space must be requested by the DCS faculty member working on the project via\u00a0<\/em><a href=\"https:\/\/resources.cs.rutgers.edu\/support\/\" target=\"_blank\" rel=\"noopener\"><em><strong>the CS HelpDesk,<\/strong><\/em><\/a><em>\u00a0<\/em>with proper details such as the project name, storage size, the NetIDs who need access, expected yearly growth, I\/O pattern, snapshots lifetime policy, etc.<\/li><\/ul><\/details><details open=\"open\"><summary><b>E. Cloud Storage<\/b><\/summary><ul><li><em><strong>Dropbox<\/strong><\/em> &#8211; Not supported. <em><em>Use Scarletmail Google Drive. You have <strong>30GB of storage space<\/strong> there. <\/em><\/em><\/li><li><em><strong>Google Drive<\/strong><\/em>.<br \/>The university has allocated 30 GB of disk space for Google Drive through the Scarlet Apps system. All CS Linux machines can be linked to Google Drive. Please see <a href=\"https:\/\/resources.cs.rutgers.edu\/resources\/connecting-google-drive-to-cs-linux-machines\/\">Connecting Google Drive with CS Linux Machines.<\/a> Note that this can only be used if you are using a Gnome graphical interface locally or via a Microsoft Remote Desktop client. Alternatively, you can use the rclone command line or <strong>rclone-browser <\/strong>GUI by following <a href=\"https:\/\/rclone.org\/drive\/\" target=\"_blank\" rel=\"noopener\">https:\/\/rclone.org\/drive\/<\/a> for configuration.<strong><br \/><\/strong>To start rclone-browser, follow:<strong><br \/><\/strong><code>export TERMINAL=\/usr\/bin\/xfce4-terminal<br \/>rclone-browser<\/code><strong><br \/><\/strong>Note: If you set mount points, set them at \/tmp\/<em>mountPointName<\/em>, where mountPointName is your chosen name. Example: <em>your_netid<\/em>-gdrive<\/li><li><em><strong>Box.com<\/strong><\/em><br \/>The University has arranged a 1TB capacity contract with Box.com for faculty\/staff. To access Box.com, please use <strong>rclone <\/strong>(command line) or <strong>rclone-browser <\/strong>(GUI) by following <a href=\"https:\/\/rclone.org\/box\/\" target=\"_blank\" rel=\"noopener\">https:\/\/rclone.org\/box\/<\/a> for configuration.<br \/>To start rclone-browser, follow:<br \/><code>export TERMINAL=\/usr\/bin\/xfce4-terminal<br \/>rclone-browser<\/code><br \/><strong>Note<\/strong>: if you set mount points up, set them at \/tmp\/mountPointName, where mountPointName is your chosen name. Example: your_netid-box<\/li><li>If there are other cloud services you want to access, contact us. Tools are available for many services, but many of them aren&#8217;t things we&#8217;d like to make generally available.<\/li><li>If you use VMs or storage in Amazon or similar environments, we&#8217;re willing to look at setting up links from systems here to them.\u00a0<\/li><\/ul><\/details><details open=\"open\"><summary><b>F. Other Storage Available<\/b><\/summary><p><strong>If you maintain your own disk space<\/strong>, would like to access it from our machines, or want to access our file systems from home computers, the following are a few options.<\/p><ul><li><strong>sshfs<\/strong><br \/>LCSR has installed sshfs on our primary clusters. You can also install it on the system you run, as well as on home systems. This allows you to mount file systems you have access to on any computer you can log in to via SSH. Performance is at least as good as an NFS mount and often better. In most cases, this is a better option than WebDAV. For details, see <a href=\"https:\/\/resources.cs.rutgers.edu\/resources\/file-storage\/accessing-files-remotely\/\">accessing files remotely<\/a>.<\/li><li><strong>\/net\/yourhost\/&#8230;<\/strong><br \/>On the faculty and research machines, we have enabled \/net automounting. So if your hostname is &#8220;myhost&#8221; and you NFS-export the filesystem &#8220;\/my\/directory\/&#8221; to the faculty or research machines, you can automount your files by connecting to &#8220;\/net\/myhost\/my\/directory&#8221;. See <a href=\"http:\/\/services.cs.rutgers.edu\/sharing.html\" target=\"_blank\" rel=\"noopener\">sharing files<\/a> for more details.<\/li><li>See our how-to page to learn more about available <a href=\"https:\/\/resources.cs.rutgers.edu\/resources\/new-users\/how-to\/#sharing\">File Sharing options<\/a>.<\/li><\/ul><\/details><\/div><p>For help with our systems or immediate assistance, visit the <a href=\"http:\/\/www.cs.rutgers.edu\/~operator\/\">LCSR Operator<\/a> at CoRE 235 or call 848-445-2443. Otherwise, see <a href=\"https:\/\/resources.cs.rutgers.edu\/support\/\" target=\"_blank\" rel=\"noopener\"><b>CS HelpDesk.<\/b><\/a> Don&#8217;t forget to include your NetID along with descriptions of your problem.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Storage Information on CS Linux Systems This page describes various storage facilities provided by LCSR for your use. However, you can use the same technology to set up your file servers or to make files on your server or desktop available from other systems. Note that LCSR is moving toward a few file servers, all [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":463,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-194","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/pages\/194","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/comments?post=194"}],"version-history":[{"count":267,"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/pages\/194\/revisions"}],"predecessor-version":[{"id":11117,"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/pages\/194\/revisions\/11117"}],"up":[{"embeddable":true,"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/pages\/463"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.rutgers.edu\/resources-cs-rutgers-edu\/wp-json\/wp\/v2\/media?parent=194"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}