a b ===== Jan. 15 - Jan. 19 ===== Advisor email: **** quoted from email **** Hi Grant, Looks good - only thing I'd say is that in the motivation, you could say that it is not only to determine whether a PCI-based CPU could work - but also what class of applications would be able to map to this. Glenn Grant Jenks wrote: > --> > > Hi Professor, > > > > Attached is my project proposal in pdf format. Let me know that this > is indeed what we discussed. I’m excited to start on it. > > > > I looked up the Mark Oskin paper and the papers that cited it and > found some useful work but nothing which would indicate this has > already been explored. I’m still looking over the results from Google > Scholar for previously done work. > > > > I’m not sure you actually have to send an email stating your > commitment to anyone. If you reply to this email affirmatively then > I’ll make sure it gets forwarded appropriately. > > > > Thanks again, > > Grant Jenks > **** end quoted message ****===== Jan 29 ===== Dear Professor, Part of the research project involves me emailing you weekly progress reports which I also copy to an online record. This is the first such progress report. This last week I understood more specifically what this research project is all about. I started doing research on the simulator that will be used. So far, from what I can tell, the PTLsim/Xen simulator system will work. I’ve been reading the user manual and will have it finished by our meeting on Wednesday. Our meeting might be a great time, I think, if you showed me the machine in your lab that I could work on with the simulation environment. My current schedule allows me to do the most research on Thursdays. Will that work? Is access to the lab restricted? I’m sure we can fill out all these questions I have on Wednesday. One new idea I thought about this week was if there should be different architectures in which different ranges of the latencies were tasted. Perhaps you already considered this but I’m starting to see many different dimensions as memory, processor, and bus latencies all affect each other. I’d like, by next week, to have the simulation environment completely set up and understood well enough, to start running the different test cases and recording/analyzing the data. This will also include identifying what applications should be tested. -Grant Jenks ===== Jan 29 ===== Dear Professor, Part of the research project involves me emailing you weekly progress reports which I also copy to an online record. This is the first such progress report. This last week I understood more specifically what this research project is all about. I started doing research on the simulator that will be used. So far, from what I can tell, the PTLsim/Xen simulator system will work. I’ve been reading the user manual and will have it finished by our meeting on Wednesday. Our meeting might be a great time, I think, if you showed me the machine in your lab that I could work on with the simulation environment. My current schedule allows me to do the most research on Thursdays. Will that work? Is access to the lab restricted? I’m sure we can fill out all these questions I have on Wednesday. One new idea I thought about this week was if there should be different architectures in which different ranges of the latencies were tasted. Perhaps you already considered this but I’m starting to see many different dimensions as memory, processor, and bus latencies all affect each other. I’d like, by next week, to have the simulation environment completely set up and understood well enough, to start running the different test cases and recording/analyzing the data. This will also include identifying what applications should be tested. -Grant Jenks ===== Feb 5 ===== Hey Professor, This week I spent time becoming familiar with 32-bit ptlsim. I downloaded, built, and ran it on the volleyball server. I’ve done two practice tests. One with the “ls” program and the other with the “touch” program. I’ve started trying to understand the options that are available and the statistics data that is generated. Furthermore, I spent a significant amount of time on Saturday trying to understand how Xen 3.0 works. I am excited to report that I’m pretty sure since Xen 3.0 can boot Windows, then ptlsim/xen should also be able to boot windows. This means that we can do our tests on an x86-64 platform in Windows with ptlsim. In order to accomplish this however, I will need a 64-bit processor with virtualization support and root access to this machine. (The Xen install is extensive.) Additionally, I have growing confidence that I can tweak the virtual processors in ptlsim to effectively simulate the proposed research. We can talk more about this on Wednesday. I have yet to be able to play with vtunes. Hopefully before Wednesday I’ll find time to download it and run some practice tests. In the more long term view, it has become clear that developing the appropriate simulator will take more than a single week. Hopefully we can get some vtunes tests working within a week or two while we determine the appropriate steps to take with ptlsim. Thanks, Grant Jenks ===== Feb 11 ===== Hey Professor, "What I've been doing..." This week I ran into a number of stumbling blocks. The computer I borrowed from you does not have virtualization support and so trying to install Xen in order to run Windows was futile. Furthermore, one of the notes in the Xen installation said that Windows only has support for one processor when it runs underneath Xen. This takes a lot of the wind out of my sails when I said previously that I had growing confidence I could make Xen/PTLsim work to accurately simulate the machine to be built. I won't be confident that this is the case until I've tried. Perhaps PTLsim works around this in some way since it has its own processor simulation. It may also be the case that PTLsim does not support Windows, period. I hope this is not the case. On the plus side I've realized that the system can be remotely managed once it is set up. This would mean I could simply remotely access a machine in your labs if we could set it up with Xen. In addition to working with the 64-bit version of PTLsim I have continued working more with the 32-bit version. For this version, I wrote my own program to test whether user interaction is supported with PTLsim. The good news is that user interaction does indeed work. The bad news is that it is very slow. I'll have to look into more optimizations that I can take advantage of in PTLsim. On the other front, I have been working with vtunes. I think vtunes is great for this project. It extensively lets us determine issues with the current hardware set up. I have also been trying to construct instances in which the vtunes manager recommends more processors being added to the system. I'll continue to run more tests in an effort to establish baselines for both the memory and the processor that might reside on the PCI board. I also will investigate the effects of turning off hyperthreading in the computer. Until we can acquire the proper hardware, this will be my primary mode of operation. "An idea I had..." The new idea I had this week is regarding vtunes. By using vtunes, the program will make suggestions on optimizations for the running programs. One of the suggestions is to obtain hardware with more processors. If we can determine the sets of programs which yield this suggestion, then we will be able to test specific programs against the new hardware configuration that is tested. This idea can potentially give focus to the project and allow for directly conclusive measurements to be made. Another set of ideas I had center around the brainstorming I did to determine what programs will be good for testing in vtunes. I have composited lists of programs which I am now combining in different ways to yield performance measurements in vtunes. Thanks, Grant Jenks ===== Feb 18 ===== Hey Professor, "What I've been doing..." This week, I worked extensively with vtune. I ran simulations under a variety of different circumstances. The first set of tests included running a processor intensive game with HyperThreading on and off and with background programs present and not present. My data so far has shown that the programs in the system tray are more or less inert and simply take up memory in the system. This prompted me to run a second set of tests in which these programs were more active in the background. I continue to experiment with these configurations as I try to develop a lower bound. I have been able to get the vtune performance analyzer to recommend upgrading to a system with more processors in several test configurations. "An idea I had..." I've got two new ideas this week. I think I can base my argument for lower bound performance metrics on the processor queue length and the number of simultaneous threads that are executing. This idea is new in the sense that I finally have a real way to quantify my argument for necessary hardware. The second idea involves a paradigm shift in my thoughts regarding the card. I had before believed this card might offer something akin to a graphics card with regards to next-generation memory. But this, I no longer believe should be the case. Instead, I think a card with a relatively large amount of cheap memory may be a better fit. Just as Windows Vista has implemented SuperFetch with usb thumb drives to added another level in the memory hierarchy, so might the gigabytes on the PCI board allow faster scratch space for data that would normally be paged out to a lower level in the memory hierarchy. Thanks, Grant Jenks ===== Feb 25 ===== Hey Professor, "What I've been doing..." This week I worked with PTLsim to try and simulate the hardware in linux. It's not that easy and I will admit that when it comes to changing the code in order to create my own processor, I'm in a bit over my head. I spent a significant amount of time tweaking parts of the core only to get confused as to what was the machine, processor, and core in the hierarchy. I'm also stuck on getting my files to build with the existing files. It's all a bit complicated. I still think I can make it happen though. I've found however that SMT/SMP machines are not supported without Xen. This means I'll have to install another operating system and become real familiar with xen and such. Since we're doing the tests now in linux, I think I may be able to use your computer after all. In addition to all this learning I ran 24 different tests with 8 different programs on a simulated single processor in order to establish a foundational set of data regarding program run statistics. I think some real good work has been accomplished. "An idea I had..." My idea this week was to get the PTLsim program to run the shell program that I created in the operating systems class at UCLA. I had planned to use this program to start other programs as if from a regular command line all while of course having the programs execute on the simulated hardware. This idea was crushed when I found out that SMT/SMP support does not exist without the Xen hypervisor. There still is hope for the idea once the Xen platform is setup at which point this may become very useful. Thanks, Grant Jenks ===== Mar 5 ===== Hey Professor, "What I've been doing..." This week I moved ahead with trying to get PTLsim/X running on top of the 64-bit Athlon machine Professor Reinman gave me. I've been relatively successful so far. I have OpenSUSE 10.2 installed with a kernel from Xen 3.0.4. Furthermore I have patched, rebuilt, and deployed the PTLsim patched Xen kernel which allows the system emulator to have control. I have also installed PTLsim/X with hypervisor features on top of all this. I hit a snag on Sunday night though when trying to test the linux kernel that is provided with the packages. I get a kernel panic error at startup when the root drive fails to mount properly. I've been pouring over forums for hours and have tried to work around the problem with a non-sata drive but to no avail. I'm going to send an email to Matt (the developer of PTLsim) tonight and see if he can shed any light on the issue. I can show you more details regarding the error on Wednesday during our meeting. I also am still waiting on the Grad student who worked with PTLsim before. I know the architecture was different but I'm sure it'll do some good. "An idea I had..." I reference this idea above actually. The idea surrounds a workaround that I thought would be effective in overcoming the error that's got me stuck. The idea was that since the kernel was panicking when it tried to mount the root device, maybe it did not have scsi support because the drives in the machines were all sata. Additionally, many forums had raised this as a possible issue. So, by borrowing a regular drive from Professor Reinman, reformatting it, and mounting it in the operating system, I thought the computer would be able to mount this drive for sure and work from there. This however did not work. I'm continuing to try different variations as forums recommend a lot of ideas. Thanks, Grant Jenks ===== Mar 11 ===== Hey Professor, "What I've been doing..." I have tried a dozen ways to get this Xen platform to work. I have not given up. I have traced the problem from the kernel panic error to a loopback error to a memory error. I'm beginning to think something is wrong with the hardware. I have tried setting up the DomU as an image file, as a device file, and as an installed operating system. I have tried using Xen with the PTLsim enhancement and without. I have combed message boards and discussion lists for hours and gone over many google results. I will soon start trying different Xen wrappers, other than OpenSuse 10.2. "An idea I had..." I think technically, any of the dozen variations I have tried ought to count as ideas. Specifically, my latest idea is to use fedora or a BSD release to see if the problem is with suse or with the hardware. I will also try the older version of suse, 10.1, which is what is used by the creators of Xen. Matt has also told me that they are going to release an update soon (I think this week) and so I will retry some of these things with the updated tree. Thanks, Grant Jenks ===== Apr 9 ===== Hey Professor, "What I've been doing..." This week I worked extensively on trying to get ptlsim with xen to work on the new hardware. I haven't gotten everything to work but have achieved more success than on the older hardware. For some reason, OpenSUSE fails to install on the new machine. Some online bug reports indicate it may be due to disk controllers. Because of this, I have switched to using Fedora. I can currently install Fedora with Xen, update it to the newest version, and connect to the hypervisor on the machine. This is as good as before. But what I can also do now is start a virtual machine without any problems. It's very cool. I've been able to run several different versions of linux simultaneously within Fedora and have increased my familiarity and understanding of Xen. The new issue has been trying to build the ptlsim extensions into the system. This was easy on the previous machine because the OpenSuse kernel was compatible with the patched Xen. Fedora, however, looks fine until you try and start a virtual machine at which point the system freezes and on reboot, the file system is fatally corrupted. Because of this, I've been working with the kernel that is released with PTLsim which is recommended for booting. The pre-built kernel files don't work and so I've been learning how to build my own kernel files from the source which is also provided. These custom-built kernel files also do not work and have prompted me to send an email to Matt with a few questions about initial ramdisks and the grub config file. Hopefully, his response will remedy the current problems and allow me to ask him more questions about changing the ptlsim code to simulate the different hardware. "An idea I had..." You brought up a good point when you said that the simulations may not necessarily be any faster with more processors. What I realized however, in working with Xen, is that the virtual machines are granted access to different processors within the system. This means that while the individual simulations will not run faster, it should be possible to run many different simulations concurrently. I'm confident that allocating two processors for each virtual machine is more than enough and so as many as 3 other virtual systems could run concurrently. Thanks, Grant Jenks ===== Apr 16 ===== Hey Professor, "What I've been doing..." This week I continued to work on the ptlsim environment. I tried implementing the platform on the Cent OS linux distribution. This distribution is more resilient than fedora and actually installs compared to suse. Nonetheless, the environment is still not ready. I am continuing to work on it. The patching and building of kernels has produced errors that are beyond the scope of this task. I am therefore redoubling my efforts to find a distribution of linux that works naturally. I am in the process of trying Debian and SELinux as well. We'll see how that pans out. I also spent time this week analyzing some of my previous results. I included this information in my checkpoint presentation which I also spent time preparing. "An idea I had..." I haven't had any particularly great idea this week. I did think it would be nice if I could write a script that automated this process of installing linux distributions and running the compatibility tests that have currently failed every time. Such a script however would have to persist between operating systems which is not really possible to my knowledge. I had one other thought which involved modifying a boot cd to package the rebuilt xen kernel with it. This might allow me to rapidly try the compatibility tests on other machines if I can deem the problem to be a hardware issue. Hopefully, this point won't be reached. Thanks, Grant Jenks ===== Apr 23 ===== Hey Professor, "What I've been doing..." I'm still working on the ptlsim environment. I tried working with the opearting system that Xen releases in the hopes that better support for the unstable version would occur. I hit a wall in this idea though when I learned that the Xen releases don't have gcc. Furthermore, trying to get gcc is nearly impossible because of the changes that Xen makes to system libraries. After working with this for a while and trying a couple different suggestions on the internet, I've decided to go back to the Cent OS distribution. I think it holds the most promise of all the distros so far. I also tried using Debian but that was nightmarish. "An idea I had..." It has occurred to me that my experiences with Xen on many different linux distributions may make for a good article. The article might simply describe some of the practicals of rolling out both regular Xen and Unstable-Xen. Maybe an IT magazine or something would be interested in publishing it as an interest piece. This might make a good addition to whatever research I publish at the end. Thanks, Grant Jenks ===== Apr 30 ===== Hey Professor, "What I've been doing..." CentOS didn't work as I had hoped. It had the same problems as Fedora but was more crash resilient. After working with CentOS for a while I sent an email to Matt desperately seeking help and have received no reply. I was at a real low point Sunday night when I had no response and no improvement and nowhere to go. I went online and continued to research the OpenSuse bug and to look for ways around the segmentation fault. I got an idea that I describe in detail below that finally has allowed me to install OpenSuse 10.2. This is a big breakthrough because this is the same distribution that Matt uses. I have sent him another email now that is very detailed about the problems I've experienced and hopefully these will be more familiar to him. Concurrently, I have been working on setting up the environments for testing the affect of changing the number of processors in the system with Xen. For this task, I have decided to dedicate a different hard drive for each operating system. The first hard drive runs OpenSuse 10.2 and is for getting ptlsim to work. The second hard drive runs the Xen distribution and is for running Windows virtual machines. The third and last hard drive runs Fedora and is for running Linux virtual machines. I've given up on expecting one OS to do it all and believe this system will work much better. "An idea I had..." My latest idea was quite simple but a big breakthrough. I have been struggling to find a distribution that works because I had not been able to install OpenSuse 10.2 which is used by the team that produces PTLsim. Last night, I went back at it again to see if I could get OpenSuse 10.2 to work. I've been really quite desperate. It occurred to me after reading many online bug reports that the problem with the disk was the metadata and a formatting of the drive might not erase this information. A format of a drive might cause all data to be lost but not necessarily all metadata. I therefore did a "dd if=/dev/zero of=/dev/sda bs=1M" and waited. After 25 minutes I finally had a blank disk. I have now been able to install OpenSuse and get to more pertinent questions with Matt. Thanks, Grant Jenks ===== May 7 ===== Hey Professor, "What I've been doing..." I did it! Finally. I've gotten PTLsim to work with OpenSuse by building a patched linux kernel from sources provided by Matt on his site. This was after I tried to get the kernel that he provides which is prebuilt to boot for a long time. I know that my abilities to build and deploy the current kernel that works is thanks to the many hours I have previously spent learning and working with the other linux distros. It feels like a great accomplishment. I can now boot the demo test machine that Matt provides and use PTLsim to monitor cpu usage. This is a huge step. This is the biggest break through yet. I am very close now to deploying a non-demo machine which I can use for the research tests. I think this is extremely promising. I have been learning a lot more now about how to manually get Xen to work with virtual machines which I make manually. It's quite a process but very promising. My current issue is that I can start and run a fully-virtualized machine but can't connect to it. Networking in a virtualized environment is, as you can imagine, quite tricky and I've been doing a lot to learn how it works. There is no need for a fall back plan now but we may need to reconsider what can practically be used as benchmarking programs on the simulated machine. "An idea I had..." The idea that struck gold wasn't much of an idea at all. Using all the skills I've developed, I calmly and carefully took the 10 million steps needed to deploy a patched linux kernel with all the necessary support. The biggest "idea" occurred when I first booted the built linux kernel. Though the kernel booted, it did not run the demo machine correctly. I thought first to try again but decided I would look over all the emails that Matt and I exchanged to see if he had already commented on this problem. There was a brief mention he had made about including specific kernel modules with a flag in a .config file. After I modified this .config file everything worked. I was really dumbfounded by the enormous effect such a small change could make. My most recent ideas regard the material I'm learning about how to connect to a fully virtualized machine. Using either vnc or sdl, I think I can make this work. It's really tricky though and I think my current idea is to use vnc over sdl even though it's slower. Hopefully I can get networking to work in it's entirety and then simply rdesktop into the machine. We'll see if thats even possible. Thanks, Grant Jenks ===== May 14 ===== Hey Professor, "What I've been doing..." More success! I've become something of a pro when it comes to writing Xen virtual machine configuration files and PTLsim command files. I have been able to port a real machine into a paravirtual machine and can then connect to it over a console and run the PTLsim tests. This is a big improvement over the PTLvm virtual machine that is provided with PTLsim. With this ability, almost any program that can work on a regular machine can be ported into a virtual machine and run with PTLsim monitoring it. I have worked more on connecting to the virtual machine as well via a graphical interface (eg. tunneling with X). I have reduced the problem to getting networking to work in the guest virtual machines. Oddly, networking works in the primary domain but does not get instantiated in the other guest domains. I've been communication with Matt more and I'm hoping it relates to a kernel module that is currently not loaded. My previous attempts to gain a graphical interface using a fully virtualized domain were thwarted by the fact that PTLsim does not currently support such an interface and so the failure I was experiencing was completely expected. (Communicating with Matt turned me on to this fact as no where is it documented in his reference manuals.) In addition to this, I've been running test scripts which simulate the machine executing on an out-of-order core and on a sequential core. The next step is to figure out how this works with multiple cores and then figure out how to change each core individually. My dialogue with Matt and my own investigating continues in this area. "An idea I had..." I must have tried at least 15 different ways to get the real machine ported to the virtual machine. First I used the perfect installation of OpenSuse that I have on another hard drive to create paravirtualized machines based on Damn Small Linux and Open Suse. These virtualized machines failed at start up by becoming stuck in the paused state. Then, I downloaded two paravirtualized machines off the internet that were supposed to be very stable and reliable. Both of these machines failed when the kernel on the Open Suse machine failed to load them properly. Again, both of these machines failed when the kernel on the Fedora Core machine failed to load them as well. Finally, I decided to install it the "old-fashioned" way. After trying, Damn Small Linux and Fedora Core and having no luck, I minimally installed OpenSuse and was able to get it to work on the mtyrel kernel with the custom ramdisk that I developed trying to get the original OpenSuse to work. It's been an exhaustive but successful process. Thanks, Grant Jenks ===== May 21 ===== Hey Professor, "What I've been doing..." I spent a significant amount of time this week composing a rough draft of the thesis. The results and analysis sections need the most work now. I feel confident that I can fill in these missing pieces in the next two weeks. I have attached a copy of the report to this email in pdf format. Additionally, I spent time compiling the results that I've gotten in Vtune to form something a bit more coherent. More work on this is still happening as well. Aside from the thesis work, I tried endlessly to get networking to work in the virtual machine. I've become proficient in both vncviewer and tunneling X through ssh but neither of these will work when the virtual machine fails to initialize networking. After a lot of searching on the net I found out that doing networking using the bridge system doesn't work in Xen version 3.0.2 with dhcp which is what I've been trying to do. After figuring this out, I gave up on a lot of the things I was trying and tried to use the other system which involves routing and forwarding. After more research and more tweaking it still would not work. I have decided that given the short amount of time that is left, it will not be worthwhile to continue trying to get networking to work. Having given up on networking, I have applied myself to getting results in the PTLsim simulator. I have learned more about how to run simulations and can show some solid data regarding a simulation. I've also been in communication with Matt about trying to simulate the other two machines that we've planned to do. "An idea I had..." This week was full of ideas that I had hoped would get networking to work. First, I thought to use vncviewer. When this failed I thought to tunnel X over ssh. When neither of these worked, I thought to manually set up the bridged network interface. I was able to set this up but not manually configure the ethernet in the guest domain. When all this failed, I thought to use routing. When even that didn't work, I ran out of ideas. Thanks, Grant Jenks ===== May 28 ===== Hey Professor, "What I've been doing..." This week, I worked for a while to revise the language of the thesis. It's much improved but still has a bit further to go. I also worked a lot with PTLsim code to produce the models we've been going for. It's quite challenging to read and understand the thousands of lines of code that define PTLsim. (Some of which is written in assembly code.) I think I've produced a couple of good working models but could be wrong in my attempts. The data collected from the runs is pretty steady so I don't see the need to run much more than 5 trials on each simulated system. I've been in communication with Matt a lot but he's really busy this week and doesn't think he could help with details regarding SMP systems until maybe Friday. Unfortunately, SMP/SMT systems are completely undocumented in his manuals and tutorials. I have also sent an email to the graduate student that created the SMT/SMP source code but have received no response yet. My final measure this week was to post to the mailing list ptlsim-devel to see if anyone could easily show me code they've written for SMP systems. The SMT/SMP stuff is less than 6 months old and so still pretty rough in documentation. Another problem that I've run into regards dual vcpu systems. There is a documented error in machines with a lot of memory in which PTLsim fails internally after the first couple simulation cycles. The fix for this problem hasn't worked for me. I tried applying the "-reservemem 256" rule to the command line but the program doesn't recognize the command. My hunch is that something has been deployed wrong. I've posted to the mailing list about this issue too. Without this, I can't run SMP systems at all. "An idea I had..." One of the problems I faced this last week was constructing models that were similar to SMP systems without being SMP systems. I have gone with a crude approximation that involves increasing the number of threads that may run in parallel. Whether this idea has much validity to it, I'm not sure but at least it's something. Thanks, Grant Jenks ===== Jun 4 ===== Hey Professor, "What I've been doing..." This last week, I tried once more to get the multi-processing capabilities working. Unfortunately, I was unsuccessful and received no response from Matt, Hui, or the PTLsim-devel mailing list. I also tried modifying the various cache parameters we discussed. The system is not as scalable as it would appear and after changing a number of different parameters repeatedly, I could get no system that worked consistently and scaled appropriately. I have run the simulations hundreds of times now across all the hardware models with three different types of loads. We can discuss this data when we meet today. Furthermore, I worked more on the thesis. I really need feedback on that as soon as possible. "An idea I had..." This week, I faced a problem with getting the simulation to start at the same place each time after kicking off multiple programs. I wrote a nifty shell script which allowed me to efficiently do this. The idea of using a shell script to kick off programs in a specific order is fairly traditional but it is unique in the context of triggering events in multiple domains. Thanks, Grant Jenks