18 Apr 2012

AFP, No Desktop Icons: Some Correspondence

Its a bit old, but still worthwhile email chain

Hi Admin,

Thank you so much for getting back to me; I'll try some of your suggestions tonight and I'll let you know how I get on!

All the best,

Mr X.

On 3 Nov 2011, at 13:58, The Admin wrote:

Hey there

Thanks for the email. It sounds very similar to the issue I was experiencing. The connection isn't being dropped, the finder is screwing up. If you traverse that directory with terminal you should still find it connected. 

I made some changes to the server which seemed to fix it. Firstly I turned off all folder redirection as after checking user accounts I found it wasn't working anyway. 

I then went through All user accounts looking for the old folder redirector symlinks and deleted them. I also looked for all other symlinks and deleted those too. For good measure i removed all ACLs (chmod -RN) and then used server admin to set them again. Then i rebooted all the clients and the server.

What i think is happening is when you make a symlink from a remote directory to a local directory or vis versa some older API somewhere in OS X creates a new filesharing connection. As you cant connect to an AFP server with more than one set of credentials at a time i believe this caused a conflict and then the finder then tries to resolve the alias to one of the connections. Or something like that anyway. I actually don't know for certain i just know the issue has gone away now and I didn't have to rebuild the server!


On 2 Nov 2011, at 13:52, Mr X wrote:
Hi there,

Hope you don't me e-mailing you, please feel free to ignore me if this is an imposition. I've been having a problem with network account users on and randomly finding all desktop icons are gone, after much googling I happened upon your blog and your post about the Carbon API Kerfuffle. The issue is seemingly fixed by a restart so I can't see any need to run a combo update every night (and I don't have a task server set up at the moment anyway), and although I have very recently turned on Folder Redirection the issue predates that.

After much wailing and gnashing of teeth and poking around it appears the AFP automount is seemingly losing it's connection and it cannot be re-established; if I browse to /Network/Serveand drill down to connect to the share where the user's folders are I get a Delete/Fix Alias prompt, but it's fine after a restart. Did you manage to gain any more insight into why this might be happening? 

Looks like it's going to be a weekend job of hosing then rebuilding the server, but I'm loathed to do that because it feels like I never find out the cause of the problem (not to mention I have to spend my weekend in a cold, dusty server room).

Here's a post I threw up on the Mac OSX Server Admin mailing list with a bit more info:

Hi all,

I have found the advice on this list invaluable over the last few years, I'm hoping you guys can shed some light on an issue I'm currently having.

I have a Mac Pro server on 10.6.8, 1TB boot partition, 2 x 3TB drives on RAID 1 with SoftRAID 4.0.7 for primary file storage, basic services (DNS, OD master handing out MCX prefs, AFP, SMB, SUS). I have about 15 client machines authenticated-bound to the OD, and about 50 users who hot desk between some of the client machines using pure network homes (I have a nice speedy network, so I thought they could live without the long log-ins of PHDs).

I'm getting an intermittent problem that I can't nail down, seems to happen pretty much every day to at least one client machine - when a user logs in, their desktop icons disappear; none of their files appear, none of the MCX-login mounted shares appear, not even the Macintosh HD icon, and you can't save or drag or create any files on the desktop - so it looks like a permissions error. However if they browse to their Home folder (e.g. from the sidebar), or browse to it through the automounted homefolders share all their files are there with the correct permissions. Restarting the client machine solves this problem.

The clients are a hotch potch of different generation iMacs and a couple of recent MBPs.
DNS is set up correctly and resolving properly on all machines (nslookup OK from client end)
OD and Kerberos seem to be OK - sudo changeip -checkhostname reports all clear, kinit xxxxx working from client machine
Clients bind to the OD without issue, and users with laptops and PHDs have no problems

So far I have tried:

- Deleting the space in the home folders automount - "Home Folders" to "homefolders"; I read that might cause an issue
- Enabled Guest access to the automount, I know this is standard practice but I really don't like having guest access on anything
- Trashing prefs to do with a wide variety of things on an affected client machine, one at a time - Kerberos, DirectoryService, Finder, MCX etc.
- Removed quotas from all users (was previously set to 20 GB, which should've been plenty)
- Trawled the logs in Console for anything suspicious, but can't find anything that I think is particularly relevant after some googling. Three that are fairly common and could be problematic are:

10/31/11 4:49:17 PM com.apple.launchd[1] (com.apple.SystemStarter) Failed to count the number of files in "/System/Library/StartupItems": No such file or directory
10/31/11 4:49:18 PM com.apple.launchd.peruser.1080[226] (com.apple.Kerberos.renew.plist[380]) Exited with exit code: 1
10/31/11 4:49:18 PM ServerScanner[376] Not scanning because node /LDAPv3/server.site.company is in searchPath

I've only seen one other case of what I'm experiencing, but it doesn't look like there was a resolution (and the deleting files bit doesn't tally up here, my issue appears to be random):

I'm totally stumped, and can't find a pattern to it - if I'm being honest I'm not even 100% sure whether it's a server or a client problem. Thoughts I've had as to what it might be are:

- The home folder automount is on a SoftRAID volume - perhaps this is causing the issue, though I have tried moving the home folders to a different, standard Apple volume
- The Mac Pro has both ethernet ports aggregated/bonded - could this cause problems?
- You're going to hate me for this one - as I could only find a 10.6.3 server disk (which wouldn't boot on the Mac Pro), I created my own server installer using InstaDMG and the 10.6.8 Combo Update. It installed perfectly and I've noticed no other issues with it (and I've set up a similar network at another site with no such issues using the same installer), but I guess this is what it could be too.
Many thanks in advance for any input you might have.

With best wishes,

Mr X

7 Apr 2012

Git "Server" on OS X Lion

My most popular post ever is the SVN on OS X Lion Server post. This is great, and with a few tweaks for 10.7.3 that method is still the best one around.

However I'm going to be working with a team that uses git for version control in the near future so I thought I would take the opportunity to learn all about it. I read and watched tonnes on the subject and really love the distributed nature of git. It reminds me of how bit coin works. More importantly will allow me to check in commits on the train.

Still to work efficiently in a team of git developers, or even just to have an online backup of a personal git repository it can be really useful to have a git "server". This is going to be a "pro" level tutorial which makes heavy use of SSH and terminal to get the job done. This won't be a tutorial for everyone, I'm just going to aim to write a neat guide about how to avoid some common pitfalls when configuring gitolite on OS X Lion Server. On the other hand, don't be afraid to give it a go, e-mail if you get stuck.

After reading the requirements for the impressive looking gitorious I settled on setting up gitolite with a more standard ssh managed workflow. "gitolite" is a perl app which operates over ssh. Each user authenticates with a separate private key and, due to clever editing of the ssh authorized_keys file, uses the knowledge of which key you used to control repo access. With gitolite, only the server admin needs to know the gory details of how everything works, and your workgroup members can follow simple instructions to set up a public key and send it to you with a few copy paste terminal commands. If you're interested these would be:

client:~ user$ ssh-keygen
*hit return a few times to accept default options
client:~ user$ cp ./.ssh/id_rsa.pub ./Desktop/2

The user now has a file on the desktop that they can email to you.

So on to configuring the server. First things first you need to have git installed on your server. You can either install the Xcode command line developer tools or, like I did, go for the git installer from the official git website. The package installs git to /usr/local/git/bin. To save yourself a long journey investigating setting PATHs for non interactive shells do yourself a favour and symlink those executables somewhere sensible.

server:~ admin$ sudo ln -s /usr/local/git/bin/* /usr/bin/

Once installed we need to set up a git user on our server. I opted to make the git user a local user, but a network account will work too. The user needs to be added to the server's Remote Login Group in the local directory so that it has permission to SSH in. You can either do this from workgroup manager by showing System Records in the View menu or you can remote desktop into the server and go to System Preferences > Sharing > Remote Login.

Then sshd needs configuring to accept public keys. I recommend configuring to ONLY accept public keys (or keys and kerberos) as it is much more secure. Follow my previous article on the subject.

Now we need to become the git user and download gitolite. You can either remote desktop to the server as admin and fire up terminal or SSH into the server with an admin user if you have that working. Use the following command to switch to the git user:

server:~ git$ sudo su - git

Then, following the instructions at the gitolite github:

server:~ git$ git clone git://github.com/sitaramc/gitolite
server:~ git$ cd gitolite
server:~ git$ git checkout g3

This clones a git repo and checks it out to the current directory. As gitolite is written in perl we are ready to go. To install, type

server:~ git$ ./gitolite/install

The last thing to do before we can start following the documentation for configuring g3 on the gitolite website is to send the public key we are going to use for gitolite to the server and configure it as the admin's key. I chose to generate one using the interactive ssh-keygen in terminal. I've found that gitolite will complain that the key must only be one line long when generated this way. I assumed this was due to line ending encoding. I opened it up in vim and it showed one line, but opening in TextMate showed two. I deleted the extra line and saved the key. This fixed the issue for me.

This then needs to be copied over to a world readable directory on the server. gitolite's documentation suggests /tmp. I used scp to get the file over.

workstation:~ admin$ scp ./admin.pub admin@server:/tmp/admin.pub

Then on the server, as the git user

server:~ git$ gitolite setup -pk /tmp/admin.pub

Then on your workstation:

workstation:~ admin$ git clone git@server:gitolite-admin.git

This will also work in your favourite graphical git client. (I've taken to GitBox, Thanks Dan Benjamin and John Siracusa)
Fingers crossed you now have the admin repository for gitolite checked out on your computer. If not then good luck as info for g3 seems to be scarce at the minute. Please feel free to email or tweet @osxserverblog if you find any extra info or if you get stuck.

17 Oct 2011

Getting Root on OS X

There is a myth that exists that to have a root prompt on an OS X machine you need to enable the root account by performing some manipulation of the system settings that is non standard and can potentially open your systems (and especially servers) to security risks.

Yes, by default OS X ships with the root account disabled but logging in as root is still possible by the combination of 2 privilege escalating commands. 

sudo su -

sudo (perform with privilege escalation)
su (switch user) - (and switch full shell environment)

The command will give you a root login prompt indistinguishable from logging in as root in the first place and you don't need to enable the root account. You are effectively invoking a shell with root permissions but as "su -" runs all a users login scripts and other things to do with setting up your shell environment it is a lot cleaner than issuing the similar command:

sudo /bin/bash

Going Deeper
Not every user on a system can invoke these commands. *BSD is normally set up so users in the group "wheel" can use sudo and can su -. On OS X we can use trusty Workgroup Manager to investigate whether this is also the case.

Fire up Workgroup Manager and instead of logging into a server type localhost in the login window

and connect with one of the local admin accounts' credentials.

You'll be presented with the familiar Workgroup Manager window but you should be looking at the local directory of the computer that you are on. Mac OS X uses open directory for local machine accounts and we can use this familiar tool to manipulate things about the local machines' user and group structures.

Most of the accounts and groups your computer uses to keep things running are initially hidden. You can show them by using the View menu and selecting "Show system records"
Now select the groups tab in the left side bar and you will see all the groups on the local computer. Quite a lot, huh. Now you'll notice that there is no wheel group on OS X. 

So where does sudo and su - get the information about which accounts to allow to act as root?
Some information is provided by running the command sudo visudo which lets system administrators change which users and groups can run what commands as root on the computer. visudo is a special modification of vi that has syntax checking and other cool stuff for editing the /etc/sudoers file. You should only use visudo for editing /etc/sudoers.

On Mac OS X the "sudoers" file includes these lines:
# User privilege specification
root    ALL=(ALL) ALL
%admin  ALL=(ALL) ALL

So its users in the admin group that can run commands as root. Checking out the "Administrators" group (usually group 80) in Workgroup Manager shows us that its short name is "admin" (as mentioned in the sudoers file) and it should contain all the users marked as admin accounts for your local computer. In fact if you remove a user from this group the tick next to "Allow user to administer this computer" in the accounts section of System Preferences becomes unticked!

Users with benefits 
So lets use all this information about which users can escalate to root and which can't to our admin advantage. Try adding an account to the Administrators group on the local computer. There is a drop down on the pop out tray that allows you to switch nodes in the search path.

You can make any user or group of user from any server your computer is bound to a local administrator on your computer by adding them to the local "Administrators" group!

This means that if I have a bunch of users who want to be admins of the computers they log into I can make a group for them on the server, call it "Local Admins" or something, add their server accounts into it, and then add the "Local Admins" group to the Administrators group on the workstation they use. Boom, no special permissions on the server and they have full control of that machine.

Its no fun having to visit every machine to set this up, but the good news is that there is a Terminal command we can push out via Apple Remote Desktop to add the server group of our "Local Administrators" to the "Administrators" group on every workstation you want them to have access to.

The command we'll use is dseditgroup (DirectoryService edit group)

sudo dseditgroup -o edit -a localadmins -t group admin

The above command adds the "localadmins" group to the admin group on the local computer. Note you have to use the groups short names. Also note that you don't have to specify that the "localadmins" group is coming from an Open Directory server. The command looks for the "localadmins" group in the machines' search path. For more information see the dseditgroup man page