Is there a formula or program that will let me scan a print and a color swatch and then tweak the RGB values for the color in photoshop using the RGB values from the scans so that the print color matches the swatch after the tweak?
What's your budget? If it is in the low thousands, you can use an Eye-one Photo + ProfileMaker is just the ticket. The profile editing component of the software has a capable spot color optimization feature. We use it for product work where a range of particular output colors needs to be an exact match to the swatch in question. Alternatively, you can print a large grid of possible colors and measure to see which one provides the closest match.
Assuming you are on a more limited budget, you should be able to make do with a combination of scanner and your eyes. Build a image containing a grid of color patches bracketing each pantone color. If you are using a RIP, building the grid in CMYK works, otherwise I would be tempted to work in LAB. Make a range of patches bracketing the Pantone color. In CMYK, try an initial grid using +/- 5% in each value. This create a file with 3^4 = 81 patches for each Pantone color; e.g. [C+5%, M, Y, K], [C, M, Y, K], [C-5%, M, Y, K], etc.
If you use the standard Epson driver, build the test file in LAB instead. You do not want to confound matters by shoving CMYK colors through the Epson driver or by relying on Photoshop's conversion from Pantone to your working RGB color space. Here I would do an initial pass using +/- 5 in LAB value, giving 27 total patches: [L+5, a, b], [L, a, b], [L-5, a, b], etc.).
Convert the test image to the canned Epson profile using Relative Colorimetric (not Abs. color because you do not want to add a color cast to lighter swatches). Print and scan along with the Pantone swatch. Open the file in Photoshop and sample the scanned Pantone swatch. Create a layer filled with the scanned swatch color and set the blending mode to difference. You are looking for the scanned color patch with the lowest total RGB value (nitpicker's note: what you really want is the patch with the lowest RMS error; i.e. minimize sqrt((R[patch] - R[pantone])^2 + (G[patch] - G[pantone])^2 + (B[patch] - B[pantone])^2).
If you have an older version of Photoshop on hand just build a Filter Factory filter to find the patch with the lowest difference. If you are at CS2 (I think; might have been CS) or higher, you can use the
Filter Foundry (freeware) plug-in. A quick filter to highlight all patches less than X different from the Pantone swatch is:[div class=\'codetop\']CODE[div class=\'codemain\' style=\'height:200px;white-space:pre;overflow:auto\']r=sqr((dif(r,10)*dif(r,10)) + (dif(g,20)*dif(g,20)) + (dif(b,30)*dif(b,30))) < 5 ? 255 : 0
g=0
b=0
Notes: This example assumes the RGB value of the Pantone swatch is (10, 20, 30) -- that's where those numbers come from. Substitute with the actual values your scanner returns. Any pixel less than 5 RGB units different from the Pantone color will be pure red, otherwise it will be black. Change the 5 to any other value to find colors closer or farther from the target swatch. Important: You want to perform a Gaussian blur with a radius of several pixels first to average out scanner noise.
You should also do an eyeball comparison of your selected patch to the swatch under the lighting you plan on using (D50, etc.). You do not want to get burned by a potential metamerism problem between Epson ink and the scanner lightsource. Assuming the patch looks reasonable, you can do another optimization session with finer grid points surrounding the color combination you found to be closest.
If you need help building an image containing the necessary grid points, send me an email or PM. The software we use for building profiling targets can do this easily.