How to job schedule or batch control?

Any IT system administration has a need for some automation, batch control, job scheduling or whatever you want to call it. Such can be setup with cronjobs, at-jobs or scheduled task setups, most likely on the server where the job/application must run.

MSSQL 2005 maintenance plans have the option of running off one server, but executing on another. Similar option should be present for schtasks on Windows server, but as with MSSQL i have not tried it, I have always executed everything on the local machine where the schedule is setup.

Some issues with this standard job scheduling setup will come up as you go along, say you want to know either of the following:

• How the execution went?
• What is executing right now?
• What was the standard output of a previous run?
• How long time did the previous jobs take?
• Did a job finish before a certain time or within a certain lenght of runtime?
• This job should only run if these first jobs have finished properly.

All this and more seems like valid points in any normal IT administration. Some of the things I quite often want to do is:

• Add new onetime only jobs, that need to run just once, eg. execute a script that creates a user, deletes a user, or stop/start a service, etc. etc.
• Add new permanent jobs, keeping history of changes in start time etc.
• Handle schedules of database servers, such as MSSQL, MySQL and Oracle.

More of what I would like in a batch control/job schedule system:

Must have:

• Setup applications with several jobsteps consisting of commandlines
• Jobs must be startable at certain times
• Keep track of history of executions, including time, returncodes etc.
• Jobs and applications must be run based on other application dependencies
• Timeouts and alternative actions, eg. alarms(email etc)
• Gui for monitoring batch progress
• Joboutput (standard output) must be available central

Nice to have:

• Load evaluation/weight of nodes, deciding where to send jobs.
• Failover execution if worknodes fail certain checks (node health check support)
• Eliminate the need for a central control server. Most nice would be all nodes to be aware of every other node, allow failover and pick up new nodes if they come alive again.

When looking into what systems can do this I get caught up in a mix of grid computing, load balancing and job/batch scheduling:

There are 31 pages in this section of this category.

Job scheduler
B
Batch queue
BatchMan
BatchPipes
Batchman
C
CONTROL-M
Command queue
Condor High-Throughput Computing System
Cronacle
G
Grid MP
H
IBM Houston Automated Spooling Program
I
IBM 2780/3780
IBM Tivoli Workload Scheduler
IBM Tivoli
Workload Scheduler LoadLeveler
J
Job Control Language
Job Entry
Subsystem 2/3
L
Load Sharing Facility
M
Maui Cluster Scheduler
Moab Cluster Suite
O
Open Source Job Scheduler
P
PTC
Scheduler
P cont.
Portable Batch System
R
RTDA Network Computer
Remote Job Entry
Retriever Communications
S
S-graph
SAP
Central Process Scheduling
SHARCNET
Sun Grid Engine
U
Unicenter
Autosys Job Management
X
Xgrid

Currently we are using TWS, and a homemade system which can do all we need, plus is extendable. Of course TWS is something we are forced to use, the other system would be just fine.

Of course I will be limited to open source or free systems, so I came up with these few systems I would like to try out:

• TORQUE is opensource and support available, that is nice for the enterprise. Torque is available for FreeBSD via ports, even very actively maintained.
• Sun Grid Engine is a batch queueing system implementing a superset of the
  functionality of the POSIX batch queueing framework. Also in FreeBSD ports.

On a side note, i stumbled upon a cluster admin article for unix with ssh, where cssh is suggested, but with much more in comments on http://www.debian-administration.org/.

Snort - what can you do

Taosecurity heads up on his 11th Snort report which is a good NSM read for most Snort administrators or just NSM interested IT security technician. Reading his books will also get you the idea of a NSM approach :-)

Some snips:

"How do I make Snort log sessions/flows?" It's inspiring to see such faith in Snort, but such questions indicate a certain amount of tool-fixation.

Snort can operate in two modes: active and passive. Snort can be active either inline or offline:

1. In an active, inline mode, Snort acts as an intrusion prevention system (IPS)...
2. In an active, offline mode, Snort acts as a quasi-IPS...
3. In passive, inline mode, Snort sits physically on the wire and allows all traffic to pass...
4. ... passive, offline mode... watches traffic provided by a network tap or switch SPAN port.... is the most popular...

...The following is a transcript generated from Sguil. The data was collected by a second instance of Snort running in pure Libpcap packet logging mode. The content was built using Tcpflow. The operating system fingerprinting was done by P0f...

... This very short example hints at the real power of Snort. I tend to see Snort as a pointer to activities that require additional inquiry. A Snort alert should be the beginning of an investigation, not the end.

Yes it is the NSM story, I like it of course :-)

Oh by the way, I can only agree with the problem of tool fixation. Tools does not solve problems, although many think so still. It requires much more :-) Related to this problem is mis-usage and security by belief (instead of fact) due to systems being setup and operated by "make install".

Windows scheduled tasks, backup/restore/administer

Recently I had to make an analysis of scheduled tasks on about 50 servers, mixed Windows 2000 and 2003. Some of the tasks was to be recreated on new Windows 2003 servers. Same project as the shares analysis.

I first turned to schtasks.exe which can be used for query (and create on 2003) , for example:
schtasks.exe /S server /delete /f /tn "calc"
schtasks.exe /S server /CREATE /SC ONSTART /TN "calc" /TR "command" /RU:"domain\user" /RP:pass
schtasks.exe /S server /run /tn "calc"

The query output gives information that can be parsed, eg:
schtasks /query /v /fo table
HostName TaskName Next Run Time Status Last Run Time Last Result Creator Schedule Task To Run Start In Comment Scheduled Task State Scheduled Type Start Time Start Date End Date Days Months Run As User Delete Task If Not Rescheduled Stop Task If Runs X Hours and X Mins Repeat: Every Repeat: Until: Time Repeat: Until: Duration Repeat: Stop If Still Running Idle Time Power Management

server calc Never 16:30:00, 12-12-2007 0 user At 16:30 every Mon, Tue, Wed, Thu, Fri of every week, starting 06-12-2007 C:\WINDOWS\system32\calc.exe calc.exe N/A Disabled Weekly 16:30:00 06-12-2007 N/A MONDAY,TUESDAY,WEDNESDAY,THURSDAY,FRIDAY N/A runasdomain/user Enabled 72:0 Disabled Disabled Disabled Disabled Disabled Disabled

But as schtasks.exe does not work on Windows 2000 I turned to jt.exe from Windows 2000 resource kit:
ftp://ftp.microsoft.com/reskit/win2000/jt.zip
3104f01eb01ce8b482bf895db60d7e8e jt.exe

I looked at some jt.exe examples, and created a parser in perl. The basic usage of jt.exe was pretty much limited to:
joblist from: jtbin /sm \"$myserver\" /se p
credentials: jtbin /sm \"$myserver\" /sac \"$jobname\" /gc

Here are some more examples of create commands, generated from parsing the jt.exe output:
Example of mon-fri 8-18, every minute:
schtasks.exe /create /SC WEEKLY /RI 1 /ST 08:00 /ET 18:00 /D MON,TUE,WED,THU,FRI /TN "task" /TR "cmd" /RU:domain\user
Every morning, mon-fri:
schtasks.exe /CREATE /SC Weekly /D MON,TUE,WED,THU,FRI /ST 07:00:00 /TN "task" /TR "command"

Later i found that i can patch SCHTASKS.EXE for Windows 2000 usage, and i turned out to actually work perfect. But i had already used jt.exe output for parsing, and it did do everything i needed. Here are the checksums of the files i tested patching with:

4D918C96C3306DF5F460801437BF24FC schtasks_w2k_5.1.2600.2180_patched.exe 86E33A8D9174DB2DB5001D0FD5DCFB8D schtasks_w2k3_5.1.2600.2180_orig.exe

Some of the problems i have or had while working with scheduled tasks:

Parsing more that the first trigger for a task.

How to make create a task or modify the default task property: "Stop Task If Runs X Hours and X Mins: 72:0". This is a problem if the task is created as ONSTART, but we want it to keep running for ever.

Worked around this by calling a cmd wrapper so the task it self is not running, but a wrapper which loops.

I did not try using "jt /? /sj" option which might be what I needed:

/SJ - set task's properties

Change one or more properties on the in-memory task object.

...

MaxRunTime = (in milliseconds)

...

Example: /sj command = notepad.exe Priority=idle DeleteWhenDone=1

How to make sure a schtask program is started in session 0?

That is, if a terminal service session 1 or 2 exists, the remote schtask /run command will sometimes(not always) start the program in session 1 or 2, which is not always what we want.

Only workaround was to manually logging into terminal services /console and starting task.

So this problem is not solved :-)

If the task is set for ONSTART it will of course start in session 0 if you reboot the server.

If there is a session 1 or 2, it does not work to use psexec eg. like this:

psexec \\server -i 0 -e cmd /C "schtasks.exe /RUN /TN calc"

Psexec -i 0 (default) and -i 2 works fine if it is not a scheduled task that is started:
psexec \\server -d -e calc.exe
psexec \\server -d -i 2 calc.exe

The jt /? /sj does not seem to have a property for what session a scheduled task starts in:

The property list has the form = ...

The task properties and the form of their values:

ApplicationName =
Parameters =
WorkingDirectory =
Comment =
Creator =
Priority = { Idle Normal High Realtime }
MaxRunTime = (in milliseconds)
Idle = (wait & deadline, in minutes)
Interactive = { 1 0 }
DontStartIfOnBatteries = { 1 0 }
KillIfGoingOnBatteries = { 1 0 }
RunOnlyIfLoggedOn = { 1 0 }
SystemRequired = { 1 0 }
DeleteWhenDone = { 1 0 }
Suspend = { 1 0 }
HaltOnError = { 1 0 }
StartOnlyIfIdle = { 1 0 }
KillOnIdleEnd = { 1 0 }
RestartOnIdleResume = { 1 0 }
Hidden = { 1 0 }
TaskFlags = (in decimal)

- must be surrounded by double quotes if it contains spaces
- { m/d/y TODAY }
- any integer

Case is not significant (i.e., IDLE and Idle are both legal).

Verify computers health before allowing network access

The topic will be interesting to any Windows administrator who worries about what client computers are allowed on the network. I could imagine that many people will have created their own ways of checking, for example before dhcp gives an ip, or blackholing ips if traffic or status of a machine fails checks.

With Network Access Protection (NAP) in Windows Server 2008 there is a new possibility.

Some quotes and hype:

• Administrators can enfore policies with NAP, eg. placing clients that fail requirements in quaratine(limited access) or with no access.
• Using NAP with DHCP lets you protect all NAP capable clients that get network access from DHCP including Wifi and lan computers.
• Windows XP SP3 will include NAP client software. Vista has it by default. Nap client software for XP beta 3 will XP SP2 NAP capable.
• NAP is not limited to Microsoft, the system just has to provide the NAP server with its health state. Example: missing!

To use NAP for DHCP you must perform these tasks: (Remember these are just some snips from Windows IT Pro november 2007 issue).

• Prepare environment: must have AD with one or more 2003(or 2008) DCs. Must have DHCP on a 2008 machine, eg. a member server in the domain. Open server manager and add Network Policy Server(NPS) which replaces 2003s Internet Authentication Server(IAS). etc etc
• Configure health policies: in the NPS console, configure the System Health Validator (SHV) to the client requirements you have. Configure the Health Policy options, select new and check the SHV's you want to use and if they must eg. pass all SHV checks to be considered healthy, eg. automatic update on, hotfixes installed, firewall on, etc etc. Also create a new health policy for clients to be considered non-compliant/unhealthy. etc etc
• Create network policies for NAP: in the NPS console setup Network Policies to specify what network access that will apply to eg. unhealthy clients. etc etc
• Configure DHCP for NAP: configure one group of scope options for compliant NAP clients and one scope for incompliant clients. Go to properties of the scope in the DHCP console, enable for this scope in the Network Access Protection tab.
• Enforce NAP on the client side: use the NAP client console, group policies or netsh (which has new NAP context). You can edit GPOs from Vista or 2008 Group Policy Management Console (GPMC). Start the Network Access Protection Agent service, and automatic of course. On Vista there is a mmc, napclcfg.msc. Netsh command is: netsh nap client set enforcement ID = 79617.
• Run a NAP test and check how you can notice if some clients fail. You will probably get a call from the client owner who can not get online.

Btw, Windows 2003 SP1 already had some Network Access Quarantine (NAQ) that helps administrators limit of deny connections to computers that dont comply with a companys security policies. However there are some problems with NAQ:

• Only works with VPN, leaving wifi and normal lan connections out of the game!
• NAQ is based on scripts that run on the client, which can be hard to create for every firewall or antivirus software you want to check
• After NAQ check is completed, the user can disable firewall or antivirus, it will not be detected, and level of access remains the same.

Of course NAP replaces NAQ:

NAP is essentially the replacement for Network Access Quarantine Control and the long-term solution for customers. Microsoft anticipates that partners will provide services and solutions to assist customers with the maintenance of their existing investment or the update of their networks for NAP.

For a detailed comparison of NAP with Network Access Quarantine Control in Windows Server 2003, see Network Access Protection Platform Overview.

So NAP seems like another tool in the box of Windows network administration, just like WSUS is.

When Vista?

At my work there is a rumor we will switch to Vista by the end of 2008. It might seem far away, but actually I think it is too soon!

I have tried Vista at home, but I skipped it for my XP again! And that was a machine i used for entertaining, multimedia and such. So I really fear being forced to use Vista for getting work done!! It will happen of course, but I hope it wont be soon!!

For a good laugh, read the Upgrade to Windows XP :-) Also on /.

Flickr statistics and Picnik editing

Finally Flickr added stats for pro accounts, thank you :) Flickr statistics was missing so its nice to see it in action! I would like to be able to go back in time, maybe that will come, so far its a good start!

And with Picnik picture editing in place there is no chance I am leaving Flickr anytime soon! Of course I prefer Google services for most anything else ... youtube, calendar, e-mail, documents and blogging of course :-)

Encrypted filesystems solutions

I recommed reading the monthly CRYPTO-GRAM, it always has interesting stories from real life security, and not just IT related. Well worth subscribing to. It is often long, but a very good security round up of the month!

This month CRYPTO-GRAM had some nice reflections on disk encryption. Still relevant even after so many years of one story after another where personal data is lost, this latest is no exception!

So it should be no surprise that many people and companies still dont use disk encryption in some form or the other, but it is sad.

Some quotes:

Computer security is hard. Software, computer and network security are all ongoing battles between attacker and defender. And in many cases the attacker has an inherent advantage: He only has to find one network flaw, while the defender has to find and fix every flaw.
...
There are several whole-disk encryption products on the market. I use PGP Disk's Whole Disk Encryption tool for two reasons. It's easy, and I trust both the company and the developers to write it securely. (Disclosure: I'm also on PGP Corp.'s Technical Advisory Board.)

Setup only takes a few minutes. After that, the program runs in the background. Everything works like before, and the performance degradation is negligible. Just make sure you choose a secure password -- PGP's encouragement of passphrases makes this much easier -- and you're secure against leaving your laptop in the airport or having it stolen out of your hotel room.

I am missing whole disk encryption on some of my computers, so i will look into that. On my Macbook i use Filevault.

There are other encryption programs out there. If you're a Windows Vista user, you might consider BitLocker. This program, embedded in the operating system, also encrypts the computer's entire drive. But it only works on the C: drive, so it won't help with external disks or USB tokens. And it can't be used to make encrypted zip files. But it's easy to use, and it's free. And many people like the open-source and free program, TrueCrypt. I know nothing about it.

I prefer TrueCrypt on Windows (didnt work on *nix when i tried a while back), having all sensitive data inside containers. On FreeBSD i use GEOM Based Disk Encryption (gbde) and EncFS.

An interesting twist and point to take note of, is if you are forced to type in your password. By authorities or criminals:

And some countries -- the United Kingdom, Singapore, Malaysia -- have passed laws giving police the authority to demand that you divulge your passwords and encryption keys.
...
Failing that, you can try to convince the authorities that you don't have the encryption key. This works better if it's a zipped archive than the whole disk. You can argue that you're transporting the files for your boss, or that you forgot the key long ago. Make sure the time stamp on the files matches your claim, though.
...
The best defense against data loss is to not have the data in the first place.

You really dont need to walk around with all kind of data, so dont!

IT security, determine your score of the game

I am not sure why I missed a really good post at Taosecurity, maybe it was the size of the post and me being tired when going over his blog. This post is very important when thinking about IT security, so once again I remind myself to keep reading Taosecurity, even if I am tired :-)

Anyway, some of the key viewpoints, some new to me, some not:

... don't think your security responsibilities end when the bottle is broken against the bow of the ship and it slides into the sea. You've got to keep watching to see if it sinks, if pirates attack, how the lifeboats handle rough seas, and so forth.

And there is an excellent list of suggestion for how to determine your enterprise "score of the game," and use that information to decide what you need to do differently.

Here are some headlines from the list:

1. Standard client build client-side survival test. Create multiple sacrificial systems with your standard build. Deploy a client-side testing solution on them, like a honeyclient
2. Standard client build server-side survival test. Create multiple sacrificial systems with your standard build. Deploy them as a honeynet.
3. Standard client build client-side penetration test. Conduct my recommendation penetration testing activities and time the result.
4. Standard client build server-side penetration test. Repeat number 3 with a server-side flavor.
5. Standard server build server-side penetration test. Repeat number 3 against your server build with a server-side flavor.
6. Deploy low-interactive honeynets and sinkhole routers in your (internal) network. These low-interaction systems provide a means to get some indications of what might be happening inside your network.
7. Conduct automated, sampled client host integrity assessments. Select a statistically valid subset of your clients and check them using multiple automated tools (malware/rootkit/etc. checkers) for indications of compromise.
8. Conduct automated, sampled server host integrity assessments. Self-explanatory.
9. Conduct manual, sampled client host integrity assessments. These are deep-dives of individual systems. You can think of it as an incident response where you have not had indication of an incident yet.
10. Conduct manual, sampled server host integrity assessments. Self-explanatory.
11. Conduct automated, sampled network host activity assessments. ... The idea is to let your NSM system see if any of the traffic it sees is out of the ordinary based on algorithms you provide.
12. Conduct manual, sampled network host activity assessments. This method is more likely to produce results. Here a skilled analyst performs deep individual analysis of traffic on a sample of machines (client and server, separately) to see if any indications of compromise appear.

In all of these cases, trend your measurements over
time...

Don't slip into thinking of inputs. Don't measure how many hosts
are running anti-virus. We want to measure outputs. We are not proposing new
controls.

Key phrases: manual vs. automated and server vs. client, and proactive investigation.

Most of the info has been on his blog before, but all toghether yet another great post :-)

Adminstrating what your DNS queries are: OpenDNS

It seems like an obvious win for your client network security, when it comes to visiting malicious hostnames: use an DNS server which denies certain hostnames based on some Realtime Block Lists (RBL). Similar to what can be used in parsing e-mails for spam points!

I read several places about OpenDNS, a great free DNS provider who does exactly what you would like, even with added administration to remove blacklists, see top queries etc. And they continue to improve the service and administration dashboard.

So check it out if you are administrating a client network intranet for example. Perhaps its is a bit too far using it for your servers :-)

Starting some PowerShell notes

For a while there was hype about Microsofts new scripting shell, it was referred to as Monad or MSH, now it is called PowerShell.

A good place to start is at Rob van der Woude's scripting pages:

Getting started:
Download and install Windows PowerShell 1.0 RtW and .NET Framework 2.0 RTM and the Windows PowerShell 1.0 Documentation Pack.
You'll need to uninstall older versions of PowerShell first.

...

PowerShell Links:
Windows PowerShell Quick Start

Here are some notes and tips from Windows IT Pro november issue:

Powershell uses a new set of commands called cmdlets and a new syntax.
Help: Get help with the get-help command.
CD: you can change to registry key: cd hklm:\software
Get-Alias cmdlet is gal, eg. list all aliases: gal select name, definition
Get-Command to see the many commands available, eg: get-command get*
Set-Content to write values to a file: sc c:\f.txt -value "Hi"
Get-Content to read contents of a file: gc c:\f.txt
Set-ExecutionPolicy: by default powershell can not run scripts, you can only enter commands at the command line. To enable run scripts: set-executionpolicy unrestricted
Set-PsDebug: for example step through one line at a time, set-psdebug -step
Get-Process: you can list all running processes: get-process
Get-Eventlog: for example: get-eventlog -newest 10 -logname system

I think I wont get started with Powershell for real until Windows 2008 / Exchange 2007 or similar is being used somewhere close to where i do my administration :-)