College of Engineering EECS Instructional Support Group July 24 2007 EECS Instructional Computing - Review and Plans ----------------------------------------------- Spring 2007 CONTENTS: Mission Statement Recent Improvements Current Initiatives Parellel Computing Cluster Notable Events Mission Statement ----------------- The EECS Instructional Support Group (ISG) installs and maintains networked computers that are used by EECS classes. ISG provides computer accounts for instructors and students in the Instructional labs and on Instructional servers. ISG purchases, installs and maintains application software needed for classes. ISG supports instructional labs in Cory Hall, Soda Hall and Hearst Field Annex. These are the functions in which ISG interacts with other UCB support groups: - we obtain enrollment lists from the Registrar (Student Information Services) - we synchronize our user accounts with the EECS department (IDSG) - we provide cardkey pre-authorization for our students to EECS Facilities - we bill students' voluntary printer charges to CARS - we coordinate our use of the EECS Network Node Bank with IDSG and CNS - we manage the computers in engineering labs with ESG - we manage the computers in EECS conference rooms with ESG - we manage the licenses for Synopsys/TCAD/HSPICE with the Device Group - we manage the licenses for Cadence with the BSAC group Recent Improvements ------------------- Due to fiscal limits, we did not make any major computer or infastructure purchases in FY 20006/2007. - purchased software licenses for Shake on Macs in 349 Soda (for CS198) - purchased 50 new chairs for labs in 330 Soda and 119 Cory Current Initiatives ------------------- - received grants from Google and Intel for a parallel computing cluster - requested AEG grant from SUN for 24 new worksatations in 275 Soda - purchased 32 20" LDCs to replace old CRTs in 105 Cory (EE20N, etc) - purchased 32 new USB kbds for 275 Soda (CS61B, etc) - purchased 96 new USB mice for 271/273/275/330 Soda - purchased 2 new printers to replace 3 old ones in 274 Soda Parellel Computing Cluster -------------------------- With generous grants from Google and Intel, we are planning to purchase this equipment for an Instructional Parallel Computing Cluster: - 26 Dell PowerEdge 1950 (each with 2 quad-core 2.33GHz Xeon, 8GB RAM) - a rack and a gigabit switch for the cluster internal network The cluster nodes will run Linux and the Nutch/HaPoop implementation of the GFS distributed filesystem. User applications will include a public-domain version of the map-reduce parallel processing tool, which is implemented in Java. We hope to have this operational by October 2007. Classes that will use it include CS61A (Harvey), CS198 (Garcia) and CS194 (Yelick). Notable Events -------------- See http://inst.eecs.berkeley.edu/notices.html for current events. --------------------------------------------------------------------------- Apr 6 - This server was down today The Instructional WEB server http://inst.eecs.berkeley.edu was down today because of a disk failure. We reinstalled it with a new disk by 3:30pm. WEB sites that require logins were fixed by 11pm. The server hosts most EECS class WEB sites as well as student WEB sites and our tech support references. --------------------------------------------------------------------------- Mar 20 - EECS network was down, 2:45pm-7pm A core router failed at about 2:45pm. It cut off Cory and Soda Halls from each other and from the outside. Computers in the Instructional labs could not reach most servers, which caused varying degrees of failure. For details, please see https://iris.eecs.berkeley.edu/news/2007/0320-Network_Failure_repo-1526.shtml Some computers froze so that the users could not logout. We logged those users out so that the contents of their accounts are safe. Network service was restored at 7pm, and residual effects were cleared from our computers by 10pm. --------------------------------------------------------------------------- Feb 7 - Exceed (XWindows) Security Alert X Windows server programs such as Exceed have been targets for password-sniffing and other security holes recently. If you are running an X Windows server on your computer at home or in an office on campus, please read this excellent article for instructions about using it securely: https://kb.berkeley.edu/jivekb/entry.jspa?externalID=1185 --------------------------------------------------------------------------- Jan 2 - UNIX and WEB site logins were disabled, Jan 1 & Jan 2 Logins to Instructional UNIX systems and email server were disabled from 0000 hours on Jan 1 through about 11O0 hours on Jan 2. A certificate had expired on our password server. --------------------------------------------------------------------------- Nov 24 - imail.eecs email server was down Thu & Fri The imail.eecs email server went down sometime after 10pm on Wednesday Nov 22. Email service was restored at about 6pm om Fri Nov 24. While imail.eecs was down, email that was sent to it was queued at the source and was delivered when imail.eecs accepted email again. --------------------------------------------------------------------------- Oct 9 - empty INBOX? your email is on http://imail.eecs.berkeley.edu (Oct 9 2006) We have stopped exporting /var/mail to all computers. If you use 'pine' on UNIX and find that your INBOX is empty on the Instructional computers, you probably need to update the way you read email. You can simply logon at http://imail.eecs.berkeley.edu to read and send email via the EECS Instructional email server. To set up spam filtering, run this command in your UNIX account: /share/b/adm/bin/instmail.pl You can also configure the email client of your choice to connect to the EECS Instructional email server. Please see http://inst.eecs.berkeley.edu/connecting.html#email for more information. --------------------------------------------------------------------------- Oct 9 - imail.eecs email server was down; cory.eecs was stalled Sometime after 11am on Sunday Oct 8, the imail.eecs email server crashed. The problem was corrected at 8am on Monday morning. While imail.eecs was down, email that sent to it was queued at the source and delivered when imail.eecs started accepting email again. A side effect was that logins on cory.eecs were frozen from about 11am though 11pm. This was because an NFS filesytem from imail.eecs was mounted to cory.eecs so archaic email programs can be run there. Cory.eecs itself and the license servers there were OK. --------------------------------------------------------------------------- July 31 - server downtimes: Samba, email, Synopsys/TCAD/Hspice licenses We plan to have these servers down for maintenance in the next 2 weeks: mamba.cs - Tuesday Aug 1, 11am-3pm Mamba is our Samba server, which allows Instructional UNIX home directories to be accessed from Windows accounts. cory.eecs - Friday Aug 4, 11am-3pm Cory is a general-access UNIX login server for EECS Instructional accounts, and it is the license server for Synopsys, HSPICE and TCAD that are used by all EECS users. While this server is down, these linceses will be unavailable. imail.eecs - Thursday Aug 10, 11am-3pm Imail is the Instructional email ("IMAP") server. While this server is down, you will not be able to read on Imail or send email from the computers in the Instructional labs. Incoming email will be queued for delivery once the server is back. July 27 - cory.eecs downtimes: July 26 (unintended), July 28 (planned) cory.eecs shut down for unknown reasons sometime after 9:30pm on Wednesday July 26. It was rebooted at 10:30am on July 27 and appears to be fine. cory.eecs will be down for a scheduled systems upgrade on Friday July 28 from about 1pm-6pm. cory.eecs is a general-access UNIX login server for EECS Instructional accounts, and it runs license servers for Synopsys, HSPICE and TCAD that are used by all EECS users. --------------------------------------------------------------------------- July 24 - UNIX logins were broken on July 22, 4:45pm-7:45pm On Sat July 22 for about 3 hours (4:45pm-7:45pm), login sessions on Instructional UNIX computers (http://inst.eecs.berkeley.edu/~inst/iesglabs.html) either stalled or were interrupted with error messages such as starting common desktop environment not able to find /share/b/bin/instructional.cshrc OS: Undefined variable This was caused by the crash of the "/share/b" file server, which has some startup scripts (instructional.cshrc, etc) on it. The server was rebooted at 7:45pm. --------------------------------------------------------------------------- July 17 - this WEB server will be down Friday July 21 3pm-6pm This WEB server (http://inst.eecs.berkeley.edu) will be down from 3pm-6pm on Friday July 21 do update the server software. The class WEB sites will be unavailable during that time. The instructors have been notified. Only EE100 has a scheduled lab at that time. --------------------------------------------------------------------------- July 12 - unable to use Firefox or Mozilla on UNIX? We are upgrading our SUN UNIX systems to Solaris 10, and the new versions of firefox and mozilla are incompatible with the older configuration settings. So if you get no response when you try to start firefox or mozilla, you should delete (or rename) your ~/.mozilla/firefox directory. It will be re-created the next time you run the browser. You can rename it with this UNIX command (you can delete it later): mv ~/.mozilla/firefox ~/.mozilla/firefox-old You can verify that the computer you are on is running Solaris 10 with the UNIX command: uname -a You can see a list of all of our login servers and their operating systems at http://inst.eecs.berkeley.edu/cgi-bin/clients.cgi?choice=servers. --------------------------------------------------------------------------- June 20 - Cory Hall will be closed overnight on Wednesday-Thursday The electric power in Cory Hall will be shutdown from 6pm on June 21 through 6am on June 22. Cory Hall will be closed to everyone during that time. Impact on Instructional computers: - Cory Hall labs will be down and inaccessible after noon Wednesday - These servers will be down from 4pm on June 21 through 2pm on June 22: cory.eecs, pulsar.eecs, quasar.eecs, c199.eecs, iserver1.eecs (UNIX) iserver2.eecs, kramnik.eecs (WINDOWS) NOT affected: - Instructional email, WEB and home directory servers (they are in Soda) - Instructional servers: nova.cs, star.cs, solar.cs - Instructional labs in Soda Hall - Instructional e-Academy site (msdnaa.eecs) --------------------------------------------------------------------------- June 6 - new SSH keys ("WARNING: HOST IDENTIFICATION HAS CHANGED!") We have updated our version of SSH and installed new "host keys" on the Instructional UNIX servers. You should go ahead and accept our new server host keys. If you are using your own SSH passphrase, you may also have to convert your public key on our servers from SSH2 format to OpenSSH format. Here are instructions: Host Keys --------- When you login using SSH, your local computer stores a copy of the server host key. When the hostkey changes on the server, SSH says: WARNING: HOST IDENTIFICATION HAS CHANGED! If you are using "SSH Secure Shell" on Windows, you can just answer "Yes" to the question "Do you want to save the new host key?" If you are using OpenSSH (usually on MacOSX or Linux), you may have to delete your old host key manually. In the WARNING message, look for a line such as Offending key in /Users/JohnDoe/.ssh/known_hosts:2 That tells you that your list of host keys is stored on your local computer in a file called /Users/JohnDoe/.ssh/known_hosts, and that you should delete line 2 (the old host key) from that file. Then run 'ssh' again and accept the new host key when it asks you. You can confirm that this new host key is valid by looking up that server under https://inst.eecs.berkeley.edu/~inst/hostkeys, or by asking inst@eecs.berkeley.edu. Public Keys ----------- SSH has a feature that lets you set your own password ("passphrase"), which is represented as a private key on your local computer and a matching public key on the remote comuter (ie, our server). The keys are text strings that are stored in files. Most Instructional users do not use public keys or their keys have already been converted. If your passphrase worked recently but fails to work now, it may be because your public key is in the old "SSH2" format an needs to converted to the new OpenSSH format. This will most often be the case if you are using the "SSH Secure Shell" client on Windows, which is an SSH2 version. You can convert the public and private keys in your Instructional UNIX account on by ssh'ing into torus.cs.berkeley.edu (it still accepts the old SSH2 keys) and typing /bin/ssh-keygen -X -f ./ssh2/your-ssh2.pub > .ssh/authorized_keys foreach id (`awk '/IdKey/ {print $2}' .ssh2/identification`) /bin/ssh-keygen -X -f .ssh2/$id >> .ssh/identity end (Replace "your-ssh2.pub" with the file name of the public key that you are using.) --------------------------------------------------------------------------- June 1 - computers down on Wed June 7, 7am-9am Computer services in EECS may be disrupted on Wed June 7, 7am-9am during maintenance on the main department servers. Login sessions on Instructional UNIX computers may stall while the /usr/sww file- system is offline. This may also stall the Instructional WEB server and email servers: http://inst.eecs.berkeley.edu http://imail.eecs.berkeley.edu (This work was originally planned on May 31 and was postponed.) --------------------------------------------------------------------------- Apr 18 - empty INBOX? your email is on cory.eecs.berkeley.edu Apr 16 - UNIX logins froze with "NFS server imail not responding" error (April 2006) We have stopped exporting /var/mail to all computers EXCEPT cory.eecs.berkeley.edu. If you use 'pine' on UNIX and find that your INBOX is empty on other Instructional computers, login into cory.eecs (using 'ssh') and run pine there. The prefered way for users to read email now is directly from the email server using the IMAP protocol. You can do that either by logging into our SquirrelMail WEB mail program (https://imail.eecs.berkeley.edu) or with an IMAP email client such as pine or Outlook. If your email is not configured for IMAP, you can run this command on an Instructional UNIX computer: /share/b/adm/bin/instmail.pl It will create new configuration files and folders (but it won't destroy any existing ones) so that pine will log you into the mail server and your email folders will be accessible by SquirrelMail and pine. Please see http://inst.eecs.berkeley.edu/connecting.html#email for some details. We used to export the /var/mail filesystem from imail.eecs to all of the Instructional UNIX computers, so that users' raw email folders there were accessible as UNIX files. This was needed for older email programs such as mailx and pine without IMAP, but now it is as unnecessary as having a file sharing connection to a WEB server so you can use an editor instead of a WEB browser to read the files. We are deprecating the "/var/mail" filesystem because if its server goes down, ALL of our UNIX login sessions to freeze while waiting for it to come back. This is a bad side effect of the UNIX NFS file sharing protocol. For example, our Instructional email server "imail.eecs.berkeley.edu" was down from about 4pm-5:30pm on April 16. (The email server had been overwhelmed by large volumes of mail which caused it to run out of virtual memory and crash.) We were still exporting /var/mail from imail.eecs to all of our UNIX systems, so during this time, all UNIX login sessions were frozen, even though most of the users were not even trying to read their email! After that incident, we decided that it was time to remove this risk and to teach our users to use the IMAP mail programs instead. We have left cory.eecs as the only computer that still has /var/mail. That foes enable legacy programs that are used to notify you of new email when you login. Those programs no longer work on our other UNIX computers. --------------------------------------------------------------------------- Apr 14 - Intermittent Network outages this week Faulty network equipment caused intermittent outages in EECS from Apr 11 - Apr 13. Network interruptions to the servers cause symtoms such as delayed responses to commands you type, loss of access to your home directory and possibly frozen login sessions. For more symptoms, see below "Symptoms when UNIX email or home directories are missing". The EECS netork staff reported on April 11: Late this afternoon our core router in Soda Hall had a hard failure on one of the management switch modules. When the router tried to fail over to the backup module, it also, exhibited a failure. I have rebooted the first module and network service has been restored. However, there may still be a problem and we are addressing this now with the vendor. Further outages may occur without warning until we get this resolved. I regret the inconvenience this may cause. This announcement, and any updates, are available at: https://iris.eecs.berkeley.edu/news/ --------------------------------------------------------------------------- Mar 20 - Why this WEB server is slow today We have detected an extreme number of hits on the http://inst.eecs WEB server from a site in India today. It appears to be downloading a specific course WEB site. Although it is apparently not malicious, it has caused delays of several seconds for some WEB pages to be displayed. We will cut off access from that site if needed. --------------------------------------------------------------------------- Feb 28 - Cory & Soda Hall power failure, 8am-9am Cory and Soda Halls lost electrical power from about 8am-9am. All Instructional computers were back up by 9:45am. Instructional printing was resumed at about 12:45pm. --------------------------------------------------------------------------- Spring 2006 - Printer usage quotas are now in effect Instructional UNIX and Windows accounts now have page limits on the Instructional shared printers in Soda, Cory and Hearst Field Annex. The print quotas this semester are computed at 50 pages for each course credit that the account is being used for. In addition, students who are EE or CS undergraduate of graduate majors are given an additional 50 pages per semster on their print quotas. The print quotas are reset each semester. Unused pages will NOT be credited to you in future semesters. If you exceed your print quota, your next print job will be replaced with a "QUOTA EXCEEDED" page. That page explains that you can logon to http://inst.eecs.berkeley.edu/webacct to view your print quota allocation and to purchase additional pages. Pages that you purchase will be billed to you, at a rate of $12 (non-refundable) for 200 pages. Reference: http://inst.eecs.berkeley.edu/cgi-bin/pub.cgi?file=html/printquota-sample.html --------------------------------------------------------------------------- Spring 2006 - Forgot your password on the Instructional computers? For named accounts: Login as 'newacct' (password 'newacct') again (in 199 Cory, 273 Soda or 'ssh' to cory.eecs.berkeley.edu). Enter your Student ID number. Then select the new "p" option for resetting your password and reprinting a form. The form will be printed the next weekday and will be available in 253 Cory after 1pm. Named accounts look like 'gbush' or 'bclinton'. For class accounts: Go to the Instructional sys admin staff in 333 Soda, 378 Cory or 386 Cory. Bring your initial class account form or student ID card. There is no on-line procedure for resetting the password of a class account. Class accounts look like 'cs61a-aa' or 'ee141-agore'. --------------------------------------------------------------------------- Fall 2004 - Security alert: Internet Explorer Active scripting is disabled Users of Internet Explorer may be prompted repeatedly for permission to run scripts on the page. This is because the sys admins have disabled the "Active scripting" feature of Internet Explorer on most EECS Instructional Windows computers because of a security flaw for which there is no patch yet. Users are advised to use a different WEB browser (Mozilla, Firefox or Opera) to avoid this. For information about the security flaw, please see http://www.kb.cert.org/vuls/id/842160 --------------------------------------------------------------------------- May 2002 - your LOST+FOUND directory Mamba.cs, the Instructional UNIX home directory server, crashed on April 29 and was down for 33 hours. For an explanation of the circumstances, please see the Spring 2002 Managers' Report (https://inst.eecs.berkeley.edu/~inst/reports/?file=Spring_2002). A number of files and home directories were not restored to their proper names and locations. When we couldn't determine where the files should go, we put them a directory called LOST+FOUND in your UNIX home directory. Please see that file for an explanation of how you can read these files, and ask us for help (inst@eecs.berkeley.edu) if needed. --------------------------------------------------------------------------- Symptoms when UNIX email or home directories are missing: - when you try to login the screen freezes - you see the error message "home directory is /" - session hangs up if you try to 'ssh' into an Instructional computer - unable to read WEB pages from the http://inst.eecs.berkeley.edu - lots of annoying "NFS timeout" error messages on your screen - new email deliveries will be delayed on imail.eecs While the server is down, you may not be able to logout in our labs because you can't type any commands. On a SunRay, even turning it off doesn't log you out. The support staff check the labs after events like this to be sure everyone gets logged out. We also post information about the problem at http://inst.eecs.berkeley.edu to help students find out when the problem has been fixed. So all you can really do in this case is to wait until the problem is fixed, go back to the lab (or login to the SunRay server for that lab) and log yourself out, or let us log you out. We disable email receipt and relaying through imail.eecs when the home directory server (mamba.cs.berkeley.edu) is down. No mail is lost. Computers that send mail queue messages that are not accepted by a remote server, and they resend the messages periodically until they are received. --------------------------------------------------------------------------- For additional information, please contact me: Kevin Mullally, ISG Manager EECS Instructional Support Group 378 Cory Hall, (510) 643-6141 kevinm@eecs.berkeley.edu http://inst.eecs.berkeley.edu/ source: ~inst/public_html/reports/managers/Spring_2007